[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260202 15:06:37.790355 11451 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.46.254:37841
I20260202 15:06:37.791685 11451 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:06:37.792533 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:06:37.807852 11460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:06:37.807955 11458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:37.809234 11451 server_base.cc:1060] running on GCE node
W20260202 15:06:37.808710 11457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:38.892006 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:06:38.892320 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:06:38.892537 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044798892514 us; error 0 us; skew 500 ppm
I20260202 15:06:38.893790 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:06:38.900678 11451 webserver.cc:492] Webserver started at http://127.11.46.254:37821/ using document root <none> and password file <none>
I20260202 15:06:38.902002 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:06:38.902285 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:06:38.902909 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:06:38.912181 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/master-0-root/instance:
uuid: "c0393909e56d448daae90116c1c00daf"
format_stamp: "Formatted at 2026-02-02 15:06:38 on dist-test-slave-rflp"
I20260202 15:06:38.920086 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260202 15:06:38.925119 11466 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:38.926098 11451 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260202 15:06:38.926407 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/master-0-root
uuid: "c0393909e56d448daae90116c1c00daf"
format_stamp: "Formatted at 2026-02-02 15:06:38 on dist-test-slave-rflp"
I20260202 15:06:38.926746 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:06:38.974903 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:06:38.976600 11451 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:06:38.977062 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:06:39.060250 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.254:37841
I20260202 15:06:39.060349 11517 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.254:37841 every 8 connection(s)
I20260202 15:06:39.070101 11518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:39.091822 11518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf: Bootstrap starting.
I20260202 15:06:39.097268 11518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:39.098976 11518 log.cc:826] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf: Log is configured to *not* fsync() on all Append() calls
I20260202 15:06:39.103358 11518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf: No bootstrap required, opened a new log
I20260202 15:06:39.122304 11518 raft_consensus.cc:359] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0393909e56d448daae90116c1c00daf" member_type: VOTER }
I20260202 15:06:39.122834 11518 raft_consensus.cc:385] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:39.123096 11518 raft_consensus.cc:740] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0393909e56d448daae90116c1c00daf, State: Initialized, Role: FOLLOWER
I20260202 15:06:39.123925 11518 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0393909e56d448daae90116c1c00daf" member_type: VOTER }
I20260202 15:06:39.124464 11518 raft_consensus.cc:399] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:06:39.124780 11518 raft_consensus.cc:493] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:06:39.125087 11518 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:39.129122 11518 raft_consensus.cc:515] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0393909e56d448daae90116c1c00daf" member_type: VOTER }
I20260202 15:06:39.129698 11518 leader_election.cc:304] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0393909e56d448daae90116c1c00daf; no voters:
I20260202 15:06:39.131213 11518 leader_election.cc:290] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:06:39.131515 11521 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:39.134065 11521 raft_consensus.cc:697] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 1 LEADER]: Becoming Leader. State: Replica: c0393909e56d448daae90116c1c00daf, State: Running, Role: LEADER
I20260202 15:06:39.134956 11521 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0393909e56d448daae90116c1c00daf" member_type: VOTER }
I20260202 15:06:39.135967 11518 sys_catalog.cc:565] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:06:39.144274 11522 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0393909e56d448daae90116c1c00daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0393909e56d448daae90116c1c00daf" member_type: VOTER } }
I20260202 15:06:39.144322 11523 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0393909e56d448daae90116c1c00daf. Latest consensus state: current_term: 1 leader_uuid: "c0393909e56d448daae90116c1c00daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0393909e56d448daae90116c1c00daf" member_type: VOTER } }
I20260202 15:06:39.145171 11522 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [sys.catalog]: This master's current role is: LEADER
I20260202 15:06:39.145231 11523 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [sys.catalog]: This master's current role is: LEADER
I20260202 15:06:39.149466 11530 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:06:39.161695 11530 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:06:39.163445 11451 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260202 15:06:39.177456 11530 catalog_manager.cc:1357] Generated new cluster ID: 274a5327dfdc46759c860eb56b3f7a2c
I20260202 15:06:39.177697 11530 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:06:39.206314 11530 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:06:39.208496 11530 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:06:39.229736 11530 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf: Generated new TSK 0
I20260202 15:06:39.230805 11530 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:06:39.294454 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:06:39.302397 11540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:06:39.303356 11541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:39.307571 11451 server_base.cc:1060] running on GCE node
W20260202 15:06:39.308421 11543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:39.309348 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:06:39.309530 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:06:39.309751 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044799309710 us; error 0 us; skew 500 ppm
I20260202 15:06:39.310273 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:06:39.314214 11451 webserver.cc:492] Webserver started at http://127.11.46.193:35859/ using document root <none> and password file <none>
I20260202 15:06:39.314836 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:06:39.315032 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:06:39.315321 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:06:39.316308 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-0-root/instance:
uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
format_stamp: "Formatted at 2026-02-02 15:06:39 on dist-test-slave-rflp"
I20260202 15:06:39.321389 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260202 15:06:39.325577 11548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:39.326346 11451 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:06:39.326596 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-0-root
uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
format_stamp: "Formatted at 2026-02-02 15:06:39 on dist-test-slave-rflp"
I20260202 15:06:39.326891 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:06:39.353246 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:06:39.354544 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:06:39.357092 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:06:39.361330 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:06:39.361590 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:39.361851 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:06:39.362064 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:39.455459 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.193:43249
I20260202 15:06:39.455520 11610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.193:43249 every 8 connection(s)
I20260202 15:06:39.462543 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:06:39.473249 11615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:06:39.475320 11616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:06:39.480674 11618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:39.481597 11451 server_base.cc:1060] running on GCE node
I20260202 15:06:39.482604 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:06:39.482906 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:06:39.483122 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044799483102 us; error 0 us; skew 500 ppm
I20260202 15:06:39.483942 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:06:39.487291 11451 webserver.cc:492] Webserver started at http://127.11.46.194:37345/ using document root <none> and password file <none>
I20260202 15:06:39.487687 11611 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37841
I20260202 15:06:39.487857 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:06:39.488081 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:06:39.488131 11611 heartbeater.cc:461] Registering TS with master...
I20260202 15:06:39.488425 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:06:39.489248 11611 heartbeater.cc:507] Master 127.11.46.254:37841 requested a full tablet report, sending...
I20260202 15:06:39.489967 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-1-root/instance:
uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
format_stamp: "Formatted at 2026-02-02 15:06:39 on dist-test-slave-rflp"
I20260202 15:06:39.492589 11483 ts_manager.cc:194] Registered new tserver with Master: 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249)
I20260202 15:06:39.495535 11483 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43272
I20260202 15:06:39.496691 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260202 15:06:39.502019 11623 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:39.502936 11451 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260202 15:06:39.503212 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-1-root
uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
format_stamp: "Formatted at 2026-02-02 15:06:39 on dist-test-slave-rflp"
I20260202 15:06:39.503455 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:06:39.519162 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:06:39.520180 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:06:39.522079 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:06:39.524716 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:06:39.524976 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:39.525188 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:06:39.525341 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:39.621373 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.194:46459
I20260202 15:06:39.621878 11686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.194:46459 every 8 connection(s)
I20260202 15:06:39.631512 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:06:39.639955 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:06:39.642534 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:39.646809 11687 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37841
I20260202 15:06:39.647296 11687 heartbeater.cc:461] Registering TS with master...
I20260202 15:06:39.648524 11687 heartbeater.cc:507] Master 127.11.46.254:37841 requested a full tablet report, sending...
I20260202 15:06:39.650204 11451 server_base.cc:1060] running on GCE node
W20260202 15:06:39.651142 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:39.651270 11483 ts_manager.cc:194] Registered new tserver with Master: 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:39.652281 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:06:39.652542 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:06:39.652702 11483 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43276
I20260202 15:06:39.652738 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044799652723 us; error 0 us; skew 500 ppm
I20260202 15:06:39.653672 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:06:39.656502 11451 webserver.cc:492] Webserver started at http://127.11.46.195:35899/ using document root <none> and password file <none>
I20260202 15:06:39.657013 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:06:39.657217 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:06:39.657471 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:06:39.658563 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-2-root/instance:
uuid: "1fcf043552924663b3127821864eb1e9"
format_stamp: "Formatted at 2026-02-02 15:06:39 on dist-test-slave-rflp"
I20260202 15:06:39.663640 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260202 15:06:39.666903 11698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:39.667765 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260202 15:06:39.668069 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-2-root
uuid: "1fcf043552924663b3127821864eb1e9"
format_stamp: "Formatted at 2026-02-02 15:06:39 on dist-test-slave-rflp"
I20260202 15:06:39.668373 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1770044797646468-11451-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:06:39.688418 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:06:39.689606 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:06:39.691315 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:06:39.694029 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:06:39.694249 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:39.694499 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:06:39.694675 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:39.784673 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.195:44527
I20260202 15:06:39.784804 11760 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.195:44527 every 8 connection(s)
I20260202 15:06:39.801486 11761 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37841
I20260202 15:06:39.801924 11761 heartbeater.cc:461] Registering TS with master...
I20260202 15:06:39.802626 11761 heartbeater.cc:507] Master 127.11.46.254:37841 requested a full tablet report, sending...
I20260202 15:06:39.804713 11483 ts_manager.cc:194] Registered new tserver with Master: 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:39.805251 11451 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012329718s
I20260202 15:06:39.806340 11483 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43290
I20260202 15:06:39.851429 11483 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43304:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:06:39.855895 11483 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:06:39.968556 11726 tablet_service.cc:1505] Processing CreateTablet for tablet 1a70ec9fde0b4e8a896aed629584f8cb (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260202 15:06:39.971532 11726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a70ec9fde0b4e8a896aed629584f8cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:39.974491 11725 tablet_service.cc:1505] Processing CreateTablet for tablet 5e50d403143c44cbb571fb0fae1a0d4e (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260202 15:06:39.974491 11724 tablet_service.cc:1505] Processing CreateTablet for tablet 9b4a87d39cb14268991cb083677738d2 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260202 15:06:39.979219 11722 tablet_service.cc:1505] Processing CreateTablet for tablet 21c950e2b059499a87376daacd8a7c14 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260202 15:06:39.978868 11723 tablet_service.cc:1505] Processing CreateTablet for tablet dd69d5f12dd2412089343e659c775418 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260202 15:06:39.981784 11725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e50d403143c44cbb571fb0fae1a0d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:39.985387 11723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd69d5f12dd2412089343e659c775418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:39.987654 11724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b4a87d39cb14268991cb083677738d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:39.992017 11649 tablet_service.cc:1505] Processing CreateTablet for tablet 9b4a87d39cb14268991cb083677738d2 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260202 15:06:39.994894 11649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b4a87d39cb14268991cb083677738d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:39.995651 11722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c950e2b059499a87376daacd8a7c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:39.989001 11651 tablet_service.cc:1505] Processing CreateTablet for tablet 1a70ec9fde0b4e8a896aed629584f8cb (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260202 15:06:39.999773 11647 tablet_service.cc:1505] Processing CreateTablet for tablet 21c950e2b059499a87376daacd8a7c14 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260202 15:06:40.005434 11648 tablet_service.cc:1505] Processing CreateTablet for tablet dd69d5f12dd2412089343e659c775418 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260202 15:06:40.011767 11651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a70ec9fde0b4e8a896aed629584f8cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.005367 11721 tablet_service.cc:1505] Processing CreateTablet for tablet fdf8b5e061f54d348072cddc0fbc2cda (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260202 15:06:40.001480 11650 tablet_service.cc:1505] Processing CreateTablet for tablet 5e50d403143c44cbb571fb0fae1a0d4e (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260202 15:06:40.013917 11646 tablet_service.cc:1505] Processing CreateTablet for tablet fdf8b5e061f54d348072cddc0fbc2cda (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260202 15:06:40.017634 11721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdf8b5e061f54d348072cddc0fbc2cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.016669 11648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd69d5f12dd2412089343e659c775418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.019145 11645 tablet_service.cc:1505] Processing CreateTablet for tablet 90f30fdf5d624adda37324efe532a47d (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260202 15:06:40.034327 11576 tablet_service.cc:1505] Processing CreateTablet for tablet 1a70ec9fde0b4e8a896aed629584f8cb (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260202 15:06:40.032979 11646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdf8b5e061f54d348072cddc0fbc2cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.036417 11720 tablet_service.cc:1505] Processing CreateTablet for tablet 90f30fdf5d624adda37324efe532a47d (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260202 15:06:40.040014 11575 tablet_service.cc:1505] Processing CreateTablet for tablet 5e50d403143c44cbb571fb0fae1a0d4e (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260202 15:06:40.040758 11574 tablet_service.cc:1505] Processing CreateTablet for tablet 9b4a87d39cb14268991cb083677738d2 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260202 15:06:40.045717 11576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a70ec9fde0b4e8a896aed629584f8cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.049537 11720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f30fdf5d624adda37324efe532a47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.049461 11572 tablet_service.cc:1505] Processing CreateTablet for tablet 21c950e2b059499a87376daacd8a7c14 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260202 15:06:40.048290 11645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f30fdf5d624adda37324efe532a47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.058053 11573 tablet_service.cc:1505] Processing CreateTablet for tablet dd69d5f12dd2412089343e659c775418 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260202 15:06:40.061980 11574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b4a87d39cb14268991cb083677738d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.069149 11644 tablet_service.cc:1505] Processing CreateTablet for tablet 28f196b3798944459a69108f9d93b0d7 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260202 15:06:40.080497 11569 tablet_service.cc:1505] Processing CreateTablet for tablet 28f196b3798944459a69108f9d93b0d7 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260202 15:06:40.078724 11575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e50d403143c44cbb571fb0fae1a0d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.078823 11647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c950e2b059499a87376daacd8a7c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.076498 11571 tablet_service.cc:1505] Processing CreateTablet for tablet fdf8b5e061f54d348072cddc0fbc2cda (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260202 15:06:40.091598 11568 tablet_service.cc:1505] Processing CreateTablet for tablet 892cc90953b1478c864856ee708597c5 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260202 15:06:40.087069 11719 tablet_service.cc:1505] Processing CreateTablet for tablet 28f196b3798944459a69108f9d93b0d7 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260202 15:06:40.095261 11719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f196b3798944459a69108f9d93b0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.095717 11650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e50d403143c44cbb571fb0fae1a0d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.095670 11573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd69d5f12dd2412089343e659c775418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.087885 11570 tablet_service.cc:1505] Processing CreateTablet for tablet 90f30fdf5d624adda37324efe532a47d (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260202 15:06:40.106634 11644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f196b3798944459a69108f9d93b0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.109431 11571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdf8b5e061f54d348072cddc0fbc2cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.114866 11569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f196b3798944459a69108f9d93b0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.122403 11568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 892cc90953b1478c864856ee708597c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.124426 11570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f30fdf5d624adda37324efe532a47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.130486 11572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c950e2b059499a87376daacd8a7c14. 1 dirs total, 0 dirs full, 0 dirs failed
W20260202 15:06:40.153216 11531 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:06:40.160701 11643 tablet_service.cc:1505] Processing CreateTablet for tablet 892cc90953b1478c864856ee708597c5 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260202 15:06:40.204764 11726 tablet_service.cc:1505] Processing CreateTablet for tablet 892cc90953b1478c864856ee708597c5 (DEFAULT_TABLE table=test-workload [id=c11807fa1f634707bec51f63b9e8863d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260202 15:06:40.205771 11726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 892cc90953b1478c864856ee708597c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.301419 11643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 892cc90953b1478c864856ee708597c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:40.313405 11780 tablet_bootstrap.cc:492] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5: Bootstrap starting.
I20260202 15:06:40.314747 11779 tablet_bootstrap.cc:492] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9: Bootstrap starting.
I20260202 15:06:40.320211 11781 tablet_bootstrap.cc:492] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71: Bootstrap starting.
I20260202 15:06:40.331359 11779 tablet_bootstrap.cc:654] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.334372 11781 tablet_bootstrap.cc:654] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.334683 11780 tablet_bootstrap.cc:654] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.340418 11779 tablet_bootstrap.cc:492] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9: No bootstrap required, opened a new log
I20260202 15:06:40.340939 11779 ts_tablet_manager.cc:1403] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.004s
I20260202 15:06:40.343266 11781 tablet_bootstrap.cc:492] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71: No bootstrap required, opened a new log
I20260202 15:06:40.343812 11781 ts_tablet_manager.cc:1403] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.007s
I20260202 15:06:40.344065 11779 raft_consensus.cc:359] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.344954 11779 raft_consensus.cc:385] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.345237 11779 raft_consensus.cc:740] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.345954 11779 consensus_queue.cc:260] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.347033 11780 tablet_bootstrap.cc:492] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5: No bootstrap required, opened a new log
I20260202 15:06:40.347455 11780 ts_tablet_manager.cc:1403] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.004s
I20260202 15:06:40.347400 11781 raft_consensus.cc:359] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:40.348516 11781 raft_consensus.cc:385] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.348924 11781 raft_consensus.cc:740] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.349800 11781 consensus_queue.cc:260] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:40.350198 11780 raft_consensus.cc:359] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:40.350996 11780 raft_consensus.cc:385] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.351260 11780 raft_consensus.cc:740] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.351892 11780 consensus_queue.cc:260] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:40.355243 11761 heartbeater.cc:499] Master 127.11.46.254:37841 was elected leader, sending a full tablet report...
I20260202 15:06:40.356249 11779 ts_tablet_manager.cc:1434] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260202 15:06:40.357087 11779 tablet_bootstrap.cc:492] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9: Bootstrap starting.
I20260202 15:06:40.360879 11780 ts_tablet_manager.cc:1434] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260202 15:06:40.361435 11611 heartbeater.cc:499] Master 127.11.46.254:37841 was elected leader, sending a full tablet report...
I20260202 15:06:40.362789 11779 tablet_bootstrap.cc:654] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.368746 11780 tablet_bootstrap.cc:492] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5: Bootstrap starting.
I20260202 15:06:40.384917 11780 tablet_bootstrap.cc:654] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.389526 11687 heartbeater.cc:499] Master 127.11.46.254:37841 was elected leader, sending a full tablet report...
I20260202 15:06:40.393635 11789 raft_consensus.cc:493] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.394129 11789 raft_consensus.cc:515] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:40.404166 11781 ts_tablet_manager.cc:1434] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent starting tablet: real 0.060s user 0.007s sys 0.006s
I20260202 15:06:40.405412 11781 tablet_bootstrap.cc:492] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71: Bootstrap starting.
I20260202 15:06:40.407591 11780 tablet_bootstrap.cc:492] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5: No bootstrap required, opened a new log
I20260202 15:06:40.408002 11780 ts_tablet_manager.cc:1403] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.000s
I20260202 15:06:40.415701 11781 tablet_bootstrap.cc:654] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.417611 11779 tablet_bootstrap.cc:492] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9: No bootstrap required, opened a new log
I20260202 15:06:40.418088 11779 ts_tablet_manager.cc:1403] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9: Time spent bootstrapping tablet: real 0.061s user 0.026s sys 0.029s
I20260202 15:06:40.419314 11789 leader_election.cc:290] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527), 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249)
I20260202 15:06:40.421516 11779 raft_consensus.cc:359] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:40.422457 11779 raft_consensus.cc:385] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.422313 11780 raft_consensus.cc:359] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.423120 11780 raft_consensus.cc:385] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.423058 11779 raft_consensus.cc:740] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.423580 11780 raft_consensus.cc:740] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.424417 11779 consensus_queue.cc:260] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:40.427145 11779 ts_tablet_manager.cc:1434] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260202 15:06:40.428164 11779 tablet_bootstrap.cc:492] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9: Bootstrap starting.
I20260202 15:06:40.430263 11780 consensus_queue.cc:260] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.437718 11780 ts_tablet_manager.cc:1434] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20260202 15:06:40.438458 11780 tablet_bootstrap.cc:492] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5: Bootstrap starting.
I20260202 15:06:40.446106 11779 tablet_bootstrap.cc:654] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.448920 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
I20260202 15:06:40.449724 11586 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 0.
I20260202 15:06:40.452364 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:40.452981 11736 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 0.
I20260202 15:06:40.454732 11624 leader_election.cc:304] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:40.455587 11789 raft_consensus.cc:2804] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:06:40.455884 11789 raft_consensus.cc:493] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:06:40.456182 11789 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:06:40.462143 11612 tablet.cc:2378] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:06:40.463501 11781 tablet_bootstrap.cc:492] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71: No bootstrap required, opened a new log
I20260202 15:06:40.464107 11781 ts_tablet_manager.cc:1403] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent bootstrapping tablet: real 0.059s user 0.024s sys 0.031s
I20260202 15:06:40.464039 11789 raft_consensus.cc:515] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:40.467926 11781 raft_consensus.cc:359] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.468788 11781 raft_consensus.cc:385] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.469090 11781 raft_consensus.cc:740] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.469826 11781 consensus_queue.cc:260] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.472605 11781 ts_tablet_manager.cc:1434] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260202 15:06:40.473573 11781 tablet_bootstrap.cc:492] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71: Bootstrap starting.
I20260202 15:06:40.480408 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:40.480782 11781 tablet_bootstrap.cc:654] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.480894 11736 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.489759 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:40.490295 11586 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.493163 11781 tablet_bootstrap.cc:492] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71: No bootstrap required, opened a new log
I20260202 15:06:40.493661 11781 ts_tablet_manager.cc:1403] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20260202 15:06:40.496381 11586 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 1.
I20260202 15:06:40.497206 11789 leader_election.cc:290] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Requested vote from peers 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527), 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249)
I20260202 15:06:40.496903 11781 raft_consensus.cc:359] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.498212 11781 raft_consensus.cc:385] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.498611 11781 raft_consensus.cc:740] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.498670 11626 leader_election.cc:304] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:40.496927 11736 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 1.
I20260202 15:06:40.499553 11781 consensus_queue.cc:260] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.501046 11786 raft_consensus.cc:2804] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:40.502302 11780 tablet_bootstrap.cc:654] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.507766 11779 tablet_bootstrap.cc:492] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9: No bootstrap required, opened a new log
I20260202 15:06:40.508162 11779 ts_tablet_manager.cc:1403] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9: Time spent bootstrapping tablet: real 0.080s user 0.045s sys 0.021s
I20260202 15:06:40.509588 11781 ts_tablet_manager.cc:1434] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20260202 15:06:40.510376 11781 tablet_bootstrap.cc:492] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71: Bootstrap starting.
I20260202 15:06:40.511052 11780 tablet_bootstrap.cc:492] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5: No bootstrap required, opened a new log
I20260202 15:06:40.510600 11779 raft_consensus.cc:359] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.511530 11780 ts_tablet_manager.cc:1403] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.002s
I20260202 15:06:40.511480 11779 raft_consensus.cc:385] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.512033 11779 raft_consensus.cc:740] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.512897 11779 consensus_queue.cc:260] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.514379 11780 raft_consensus.cc:359] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.515151 11780 raft_consensus.cc:385] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.515439 11780 raft_consensus.cc:740] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.515791 11779 ts_tablet_manager.cc:1434] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260202 15:06:40.516626 11779 tablet_bootstrap.cc:492] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9: Bootstrap starting.
I20260202 15:06:40.516103 11780 consensus_queue.cc:260] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.521493 11780 ts_tablet_manager.cc:1434] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260202 15:06:40.521445 11786 raft_consensus.cc:697] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:40.518522 11781 tablet_bootstrap.cc:654] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.522476 11786 consensus_queue.cc:237] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:40.525506 11779 tablet_bootstrap.cc:654] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.528568 11780 tablet_bootstrap.cc:492] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5: Bootstrap starting.
I20260202 15:06:40.542752 11780 tablet_bootstrap.cc:654] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.546464 11779 tablet_bootstrap.cc:492] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9: No bootstrap required, opened a new log
I20260202 15:06:40.547030 11779 ts_tablet_manager.cc:1403] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.023s
I20260202 15:06:40.561447 11781 tablet_bootstrap.cc:492] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71: No bootstrap required, opened a new log
I20260202 15:06:40.560830 11779 raft_consensus.cc:359] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.561848 11779 raft_consensus.cc:385] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.561998 11781 ts_tablet_manager.cc:1403] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.010s
I20260202 15:06:40.562191 11779 raft_consensus.cc:740] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.563479 11779 consensus_queue.cc:260] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.562418 11780 tablet_bootstrap.cc:492] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5: No bootstrap required, opened a new log
I20260202 15:06:40.565340 11780 ts_tablet_manager.cc:1403] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.008s
I20260202 15:06:40.566097 11781 raft_consensus.cc:359] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.566895 11781 raft_consensus.cc:385] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.567262 11781 raft_consensus.cc:740] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.564427 11481 catalog_manager.cc:5654] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:40.568188 11781 consensus_queue.cc:260] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.570366 11780 raft_consensus.cc:359] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.572211 11781 ts_tablet_manager.cc:1434] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent starting tablet: real 0.010s user 0.002s sys 0.005s
I20260202 15:06:40.572824 11779 ts_tablet_manager.cc:1434] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9: Time spent starting tablet: real 0.025s user 0.006s sys 0.003s
I20260202 15:06:40.573688 11780 raft_consensus.cc:385] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.574664 11781 tablet_bootstrap.cc:492] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71: Bootstrap starting.
I20260202 15:06:40.574657 11780 raft_consensus.cc:740] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.583220 11779 tablet_bootstrap.cc:492] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9: Bootstrap starting.
I20260202 15:06:40.584512 11780 consensus_queue.cc:260] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.588485 11780 ts_tablet_manager.cc:1434] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20260202 15:06:40.589577 11780 tablet_bootstrap.cc:492] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5: Bootstrap starting.
I20260202 15:06:40.596202 11781 tablet_bootstrap.cc:654] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.596203 11779 tablet_bootstrap.cc:654] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.596446 11785 raft_consensus.cc:493] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.597062 11785 raft_consensus.cc:515] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.599649 11785 leader_election.cc:290] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:40.606186 11780 tablet_bootstrap.cc:654] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.617682 11781 tablet_bootstrap.cc:492] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71: No bootstrap required, opened a new log
I20260202 15:06:40.618239 11781 ts_tablet_manager.cc:1403] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.002s
I20260202 15:06:40.621587 11781 raft_consensus.cc:359] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.622875 11787 raft_consensus.cc:493] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.624928 11787 raft_consensus.cc:515] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.626238 11779 tablet_bootstrap.cc:492] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9: No bootstrap required, opened a new log
I20260202 15:06:40.627290 11779 ts_tablet_manager.cc:1403] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.015s
I20260202 15:06:40.630677 11779 raft_consensus.cc:359] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.635090 11779 raft_consensus.cc:385] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.636334 11787 leader_election.cc:290] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.638586 11779 raft_consensus.cc:740] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.639879 11779 consensus_queue.cc:260] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.641513 11780 tablet_bootstrap.cc:492] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5: No bootstrap required, opened a new log
I20260202 15:06:40.643752 11780 ts_tablet_manager.cc:1403] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.005s
I20260202 15:06:40.644778 11786 raft_consensus.cc:493] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.645282 11786 raft_consensus.cc:515] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.647249 11785 raft_consensus.cc:493] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.647753 11785 raft_consensus.cc:515] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.648119 11786 leader_election.cc:290] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.648569 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
I20260202 15:06:40.649171 11586 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 0.
I20260202 15:06:40.649617 11785 leader_election.cc:290] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:40.651070 11779 ts_tablet_manager.cc:1434] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s
I20260202 15:06:40.651990 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:40.652747 11736 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 0.
I20260202 15:06:40.653437 11779 tablet_bootstrap.cc:492] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9: Bootstrap starting.
I20260202 15:06:40.650288 11626 leader_election.cc:304] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:40.653914 11780 raft_consensus.cc:359] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.654728 11780 raft_consensus.cc:385] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.655105 11780 raft_consensus.cc:740] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.622483 11781 raft_consensus.cc:385] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.655994 11786 raft_consensus.cc:2804] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:06:40.656437 11786 raft_consensus.cc:493] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:06:40.656917 11786 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.664347 11786 raft_consensus.cc:515] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.666486 11781 raft_consensus.cc:740] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.666729 11780 consensus_queue.cc:260] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.669651 11780 ts_tablet_manager.cc:1434] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260202 15:06:40.671205 11781 consensus_queue.cc:260] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.671701 11786 leader_election.cc:290] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.674250 11780 tablet_bootstrap.cc:492] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5: Bootstrap starting.
I20260202 15:06:40.675642 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:40.682057 11586 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.683872 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:40.684514 11736 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.687525 11781 ts_tablet_manager.cc:1434] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent starting tablet: real 0.069s user 0.009s sys 0.001s
I20260202 15:06:40.693429 11781 tablet_bootstrap.cc:492] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71: Bootstrap starting.
I20260202 15:06:40.698412 11779 tablet_bootstrap.cc:654] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.707834 11586 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 1.
I20260202 15:06:40.708452 11662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" is_pre_election: true
I20260202 15:06:40.709587 11626 leader_election.cc:304] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:40.711638 11796 raft_consensus.cc:2804] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader election won for term 1
W20260202 15:06:40.712440 11552 leader_election.cc:343] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.711771 11584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
I20260202 15:06:40.713776 11796 raft_consensus.cc:697] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:40.713104 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:40.711477 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
W20260202 15:06:40.716265 11549 leader_election.cc:343] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.715087 11796 consensus_queue.cc:237] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.716987 11585 raft_consensus.cc:2393] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fcf043552924663b3127821864eb1e9 in current term 1: Already voted for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in this term.
I20260202 15:06:40.716645 11549 leader_election.cc:304] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5; no voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:40.717850 11787 raft_consensus.cc:2749] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260202 15:06:40.719383 11701 leader_election.cc:343] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.724179 11781 tablet_bootstrap.cc:654] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.724232 11662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" is_pre_election: true
I20260202 15:06:40.723151 11736 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 1.
I20260202 15:06:40.727435 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" is_pre_election: true
I20260202 15:06:40.727546 11702 leader_election.cc:304] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcf043552924663b3127821864eb1e9; no voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:40.729265 11785 raft_consensus.cc:2749] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:06:40.734061 11780 tablet_bootstrap.cc:654] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.740792 11781 tablet_bootstrap.cc:492] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71: No bootstrap required, opened a new log
I20260202 15:06:40.741735 11781 ts_tablet_manager.cc:1403] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.009s
I20260202 15:06:40.744490 11781 raft_consensus.cc:359] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.745204 11781 raft_consensus.cc:385] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.745496 11781 raft_consensus.cc:740] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Initialized, Role: FOLLOWER
W20260202 15:06:40.745591 11702 leader_election.cc:343] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.746045 11702 leader_election.cc:304] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcf043552924663b3127821864eb1e9; no voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:40.746800 11785 raft_consensus.cc:2749] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:06:40.746876 11481 catalog_manager.cc:5654] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:40.749604 11780 tablet_bootstrap.cc:492] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5: No bootstrap required, opened a new log
I20260202 15:06:40.750062 11780 ts_tablet_manager.cc:1403] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.008s
I20260202 15:06:40.746197 11781 consensus_queue.cc:260] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.753212 11781 ts_tablet_manager.cc:1434] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20260202 15:06:40.753203 11780 raft_consensus.cc:359] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.754124 11780 raft_consensus.cc:385] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.754441 11780 raft_consensus.cc:740] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.754612 11781 tablet_bootstrap.cc:492] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71: Bootstrap starting.
I20260202 15:06:40.755184 11780 consensus_queue.cc:260] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.757503 11780 ts_tablet_manager.cc:1434] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260202 15:06:40.758639 11780 tablet_bootstrap.cc:492] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5: Bootstrap starting.
I20260202 15:06:40.762629 11781 tablet_bootstrap.cc:654] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.763928 11779 tablet_bootstrap.cc:492] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9: No bootstrap required, opened a new log
I20260202 15:06:40.764452 11779 ts_tablet_manager.cc:1403] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.005s
I20260202 15:06:40.766054 11780 tablet_bootstrap.cc:654] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.767166 11796 raft_consensus.cc:493] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.767477 11779 raft_consensus.cc:359] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.767670 11796 raft_consensus.cc:515] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.768246 11779 raft_consensus.cc:385] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.768571 11779 raft_consensus.cc:740] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.769799 11796 leader_election.cc:290] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.769261 11779 consensus_queue.cc:260] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.771857 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:40.772230 11779 ts_tablet_manager.cc:1434] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260202 15:06:40.772502 11736 raft_consensus.cc:2468] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 0.
I20260202 15:06:40.773193 11779 tablet_bootstrap.cc:492] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9: Bootstrap starting.
I20260202 15:06:40.773689 11624 leader_election.cc:304] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:40.774895 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
W20260202 15:06:40.777585 11626 leader_election.cc:343] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.779101 11779 tablet_bootstrap.cc:654] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.782526 11796 raft_consensus.cc:2804] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:06:40.783041 11796 raft_consensus.cc:493] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:06:40.783481 11796 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.795451 11787 raft_consensus.cc:493] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.795989 11787 raft_consensus.cc:515] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.797930 11787 leader_election.cc:290] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.799037 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:40.799177 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" is_pre_election: true
I20260202 15:06:40.799583 11736 raft_consensus.cc:2468] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 0.
W20260202 15:06:40.800264 11552 leader_election.cc:343] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.801170 11796 raft_consensus.cc:515] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.802923 11796 leader_election.cc:290] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.802937 11549 leader_election.cc:304] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters: 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:40.804777 11787 raft_consensus.cc:2804] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:06:40.805166 11787 raft_consensus.cc:493] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:06:40.805572 11787 raft_consensus.cc:3060] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.805706 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:40.806334 11736 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.806401 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
W20260202 15:06:40.810254 11626 leader_election.cc:343] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.812319 11787 raft_consensus.cc:515] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.814790 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:40.815335 11736 raft_consensus.cc:2468] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 1.
W20260202 15:06:40.816016 11552 leader_election.cc:343] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.816391 11624 leader_election.cc:304] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters: 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:40.817301 11796 raft_consensus.cc:2804] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:40.817415 11787 leader_election.cc:290] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.817736 11796 raft_consensus.cc:697] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:40.817811 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:40.818313 11781 tablet_bootstrap.cc:492] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71: No bootstrap required, opened a new log
I20260202 15:06:40.818354 11736 raft_consensus.cc:3060] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.818665 11781 ts_tablet_manager.cc:1403] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent bootstrapping tablet: real 0.064s user 0.013s sys 0.000s
I20260202 15:06:40.818475 11796 consensus_queue.cc:237] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.821743 11781 raft_consensus.cc:359] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.822295 11781 raft_consensus.cc:385] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.822474 11781 raft_consensus.cc:740] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.824312 11781 consensus_queue.cc:260] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.825778 11785 raft_consensus.cc:493] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.826206 11785 raft_consensus.cc:515] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.826988 11781 ts_tablet_manager.cc:1434] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260202 15:06:40.827970 11781 tablet_bootstrap.cc:492] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71: Bootstrap starting.
I20260202 15:06:40.828066 11785 leader_election.cc:290] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:40.828490 11736 raft_consensus.cc:2468] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 1.
I20260202 15:06:40.828958 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
I20260202 15:06:40.829510 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" is_pre_election: true
I20260202 15:06:40.830044 11549 leader_election.cc:304] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters: 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:40.831346 11780 tablet_bootstrap.cc:492] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5: No bootstrap required, opened a new log
W20260202 15:06:40.831625 11701 leader_election.cc:343] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.830128 11661 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 0.
I20260202 15:06:40.831840 11780 ts_tablet_manager.cc:1403] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent bootstrapping tablet: real 0.073s user 0.019s sys 0.029s
I20260202 15:06:40.833227 11779 tablet_bootstrap.cc:492] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9: No bootstrap required, opened a new log
I20260202 15:06:40.833622 11779 ts_tablet_manager.cc:1403] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.002s
I20260202 15:06:40.834678 11780 raft_consensus.cc:359] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.835388 11780 raft_consensus.cc:385] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.835659 11780 raft_consensus.cc:740] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.835907 11781 tablet_bootstrap.cc:654] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.836289 11780 consensus_queue.cc:260] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.837193 11702 leader_election.cc:304] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters: 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:40.838274 11785 raft_consensus.cc:2804] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:06:40.838627 11785 raft_consensus.cc:493] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:06:40.838853 11787 raft_consensus.cc:2804] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:40.839071 11785 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.843099 11786 raft_consensus.cc:493] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.843745 11786 raft_consensus.cc:515] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.836544 11779 raft_consensus.cc:359] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.844650 11779 raft_consensus.cc:385] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.845057 11779 raft_consensus.cc:740] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.845571 11786 leader_election.cc:290] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.845752 11779 consensus_queue.cc:260] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.846314 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdf8b5e061f54d348072cddc0fbc2cda" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
I20260202 15:06:40.846566 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdf8b5e061f54d348072cddc0fbc2cda" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:40.846932 11585 raft_consensus.cc:2468] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 0.
I20260202 15:06:40.847317 11736 raft_consensus.cc:2468] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 0.
I20260202 15:06:40.847976 11626 leader_election.cc:304] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:40.848649 11796 raft_consensus.cc:2804] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:06:40.849017 11796 raft_consensus.cc:493] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:06:40.849427 11796 raft_consensus.cc:3060] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.850878 11785 raft_consensus.cc:515] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.852516 11785 leader_election.cc:290] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:40.853466 11480 catalog_manager.cc:5654] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:40.856331 11781 tablet_bootstrap.cc:492] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71: No bootstrap required, opened a new log
I20260202 15:06:40.856714 11781 ts_tablet_manager.cc:1403] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.001s
I20260202 15:06:40.858294 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:40.858808 11661 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.859468 11787 raft_consensus.cc:697] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:40.860345 11787 consensus_queue.cc:237] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.862095 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:40.862633 11585 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.862404 11781 raft_consensus.cc:359] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.864229 11781 raft_consensus.cc:385] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:40.864818 11781 raft_consensus.cc:740] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Initialized, Role: FOLLOWER
I20260202 15:06:40.872139 11781 consensus_queue.cc:260] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.872920 11796 raft_consensus.cc:515] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.876397 11796 leader_election.cc:290] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.876691 11786 raft_consensus.cc:493] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.877976 11786 raft_consensus.cc:515] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.878885 11584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdf8b5e061f54d348072cddc0fbc2cda" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:40.879066 11780 ts_tablet_manager.cc:1434] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent starting tablet: real 0.047s user 0.012s sys 0.011s
I20260202 15:06:40.879379 11584 raft_consensus.cc:3060] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.897922 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
I20260202 15:06:40.899721 11585 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 1.
I20260202 15:06:40.901273 11701 leader_election.cc:304] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:40.903168 11786 leader_election.cc:290] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:40.904804 11780 tablet_bootstrap.cc:492] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5: Bootstrap starting.
I20260202 15:06:40.904953 11825 raft_consensus.cc:2804] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:40.907701 11661 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 1.
I20260202 15:06:40.906018 11825 raft_consensus.cc:697] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Becoming Leader. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:40.908850 11825 consensus_queue.cc:237] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.907326 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdf8b5e061f54d348072cddc0fbc2cda" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:40.913569 11779 ts_tablet_manager.cc:1434] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9: Time spent starting tablet: real 0.080s user 0.008s sys 0.018s
I20260202 15:06:40.916386 11785 raft_consensus.cc:493] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:40.916611 11779 tablet_bootstrap.cc:492] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9: Bootstrap starting.
I20260202 15:06:40.916807 11785 raft_consensus.cc:515] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.918488 11785 leader_election.cc:290] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:40.914889 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:40.919543 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" is_pre_election: true
I20260202 15:06:40.920110 11661 raft_consensus.cc:2468] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 0.
I20260202 15:06:40.920393 11735 raft_consensus.cc:2393] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9079295b8e5e40dfbc4d95468b0a0f71 in current term 1: Already voted for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in this term.
I20260202 15:06:40.921250 11702 leader_election.cc:304] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:40.921907 11785 raft_consensus.cc:2804] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:06:40.922250 11785 raft_consensus.cc:493] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:06:40.922358 11624 leader_election.cc:304] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9079295b8e5e40dfbc4d95468b0a0f71; no voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:40.922580 11785 raft_consensus.cc:3060] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.914191 11736 raft_consensus.cc:3060] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.928100 11781 ts_tablet_manager.cc:1434] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent starting tablet: real 0.071s user 0.006s sys 0.001s
I20260202 15:06:40.926714 11786 raft_consensus.cc:3060] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.934908 11781 tablet_bootstrap.cc:492] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71: Bootstrap starting.
I20260202 15:06:40.937434 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
W20260202 15:06:40.949375 11701 leader_election.cc:343] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.929901 11584 raft_consensus.cc:2468] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 1.
I20260202 15:06:40.951129 11626 leader_election.cc:304] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:40.952147 11785 raft_consensus.cc:515] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.955180 11796 raft_consensus.cc:2804] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:40.955514 11785 leader_election.cc:290] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:40.955821 11796 raft_consensus.cc:697] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:40.954171 11482 catalog_manager.cc:5654] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193). New cstate: current_term: 1 leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:40.956598 11796 consensus_queue.cc:237] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.956576 11780 tablet_bootstrap.cc:654] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.961359 11584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
W20260202 15:06:40.962487 11701 leader_election.cc:343] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:06:40.956911 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:40.965965 11781 tablet_bootstrap.cc:654] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.970607 11661 raft_consensus.cc:3060] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:40.971702 11736 raft_consensus.cc:2468] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 1.
I20260202 15:06:40.969697 11480 catalog_manager.cc:5654] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fcf043552924663b3127821864eb1e9 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1fcf043552924663b3127821864eb1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: HEALTHY } } }
I20260202 15:06:40.984732 11786 raft_consensus.cc:2749] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:06:40.982466 11661 raft_consensus.cc:2468] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 1.
I20260202 15:06:40.986275 11702 leader_election.cc:304] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters: 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:40.987157 11785 raft_consensus.cc:2804] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:40.987564 11785 raft_consensus.cc:697] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Becoming Leader. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:40.988226 11785 consensus_queue.cc:237] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:40.994753 11779 tablet_bootstrap.cc:654] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:40.996640 11780 tablet_bootstrap.cc:492] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5: No bootstrap required, opened a new log
I20260202 15:06:40.997144 11780 ts_tablet_manager.cc:1403] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent bootstrapping tablet: real 0.093s user 0.017s sys 0.014s
I20260202 15:06:40.997969 11479 catalog_manager.cc:5654] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:41.000022 11781 tablet_bootstrap.cc:492] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71: No bootstrap required, opened a new log
I20260202 15:06:41.000499 11781 ts_tablet_manager.cc:1403] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.005s
I20260202 15:06:41.003271 11781 raft_consensus.cc:359] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.003973 11781 raft_consensus.cc:385] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:41.004271 11781 raft_consensus.cc:740] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Initialized, Role: FOLLOWER
I20260202 15:06:41.005447 11781 consensus_queue.cc:260] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.008692 11781 ts_tablet_manager.cc:1434] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260202 15:06:41.009718 11780 raft_consensus.cc:359] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.010677 11780 raft_consensus.cc:385] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:41.011005 11780 raft_consensus.cc:740] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Initialized, Role: FOLLOWER
I20260202 15:06:41.011596 11780 consensus_queue.cc:260] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.013882 11780 ts_tablet_manager.cc:1434] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent starting tablet: real 0.016s user 0.002s sys 0.003s
I20260202 15:06:41.014815 11780 tablet_bootstrap.cc:492] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5: Bootstrap starting.
I20260202 15:06:41.019106 11779 tablet_bootstrap.cc:492] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9: No bootstrap required, opened a new log
I20260202 15:06:41.019462 11779 ts_tablet_manager.cc:1403] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9: Time spent bootstrapping tablet: real 0.103s user 0.007s sys 0.021s
I20260202 15:06:41.021713 11779 raft_consensus.cc:359] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.022306 11779 raft_consensus.cc:385] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:41.021562 11483 catalog_manager.cc:5654] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fcf043552924663b3127821864eb1e9 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1fcf043552924663b3127821864eb1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: HEALTHY } } }
I20260202 15:06:41.022539 11779 raft_consensus.cc:740] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Initialized, Role: FOLLOWER
I20260202 15:06:41.023193 11779 consensus_queue.cc:260] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.028131 11779 ts_tablet_manager.cc:1434] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260202 15:06:41.028796 11779 tablet_bootstrap.cc:492] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9: Bootstrap starting.
I20260202 15:06:41.029156 11780 tablet_bootstrap.cc:654] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:41.034929 11779 tablet_bootstrap.cc:654] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:41.034739 11786 consensus_queue.cc:1048] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:06:41.046290 11822 raft_consensus.cc:493] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:41.046917 11822 raft_consensus.cc:515] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.048734 11779 tablet_bootstrap.cc:492] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9: No bootstrap required, opened a new log
I20260202 15:06:41.048974 11822 leader_election.cc:290] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.049156 11779 ts_tablet_manager.cc:1403] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.012s
I20260202 15:06:41.049948 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892cc90953b1478c864856ee708597c5" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" is_pre_election: true
I20260202 15:06:41.050630 11661 raft_consensus.cc:2468] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 0.
I20260202 15:06:41.051236 11779 raft_consensus.cc:359] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.051784 11779 raft_consensus.cc:385] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:41.052040 11779 raft_consensus.cc:740] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Initialized, Role: FOLLOWER
I20260202 15:06:41.051932 11552 leader_election.cc:304] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:41.053438 11736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892cc90953b1478c864856ee708597c5" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:41.055382 11822 raft_consensus.cc:2804] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:06:41.055351 11779 consensus_queue.cc:260] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.055825 11822 raft_consensus.cc:493] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:06:41.056242 11822 raft_consensus.cc:3060] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:41.056489 11736 raft_consensus.cc:2468] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 0.
I20260202 15:06:41.057858 11779 ts_tablet_manager.cc:1434] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260202 15:06:41.065153 11789 consensus_queue.cc:1048] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.066876 11822 raft_consensus.cc:515] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.068514 11822 leader_election.cc:290] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.069288 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892cc90953b1478c864856ee708597c5" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:41.069772 11661 raft_consensus.cc:3060] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:41.072038 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892cc90953b1478c864856ee708597c5" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:41.072664 11734 raft_consensus.cc:3060] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:41.076768 11661 raft_consensus.cc:2468] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 1.
I20260202 15:06:41.079507 11552 leader_election.cc:304] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:41.080236 11822 raft_consensus.cc:2804] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:41.080647 11822 raft_consensus.cc:697] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:41.081394 11822 consensus_queue.cc:237] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.083559 11734 raft_consensus.cc:2468] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 1.
I20260202 15:06:41.087472 11780 tablet_bootstrap.cc:492] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5: No bootstrap required, opened a new log
I20260202 15:06:41.087914 11780 ts_tablet_manager.cc:1403] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.019s
I20260202 15:06:41.090229 11780 raft_consensus.cc:359] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.090656 11780 raft_consensus.cc:385] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:41.090955 11780 raft_consensus.cc:740] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Initialized, Role: FOLLOWER
I20260202 15:06:41.091589 11780 consensus_queue.cc:260] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.093168 11780 ts_tablet_manager.cc:1434] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:06:41.099159 11483 catalog_manager.cc:5654] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193). New cstate: current_term: 1 leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:41.124861 11822 raft_consensus.cc:493] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:41.125344 11822 raft_consensus.cc:515] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.127189 11822 leader_election.cc:290] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.127765 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" is_pre_election: true
I20260202 15:06:41.128093 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:41.128831 11735 raft_consensus.cc:2393] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in current term 1: Already voted for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in this term.
I20260202 15:06:41.129618 11549 leader_election.cc:304] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5; no voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:41.130288 11822 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:41.134064 11822 raft_consensus.cc:2749] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260202 15:06:41.156883 11531 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:06:41.179200 11822 raft_consensus.cc:493] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:41.179551 11822 raft_consensus.cc:515] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.180894 11822 leader_election.cc:290] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.181910 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" is_pre_election: true
I20260202 15:06:41.181910 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:41.182771 11661 raft_consensus.cc:2393] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in current term 1: Already voted for candidate 1fcf043552924663b3127821864eb1e9 in this term.
I20260202 15:06:41.183774 11552 leader_election.cc:304] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5; no voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:41.184453 11822 raft_consensus.cc:3060] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:41.188833 11822 raft_consensus.cc:2749] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:06:41.201696 11789 consensus_queue.cc:1048] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.217628 11789 consensus_queue.cc:1048] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.238075 11789 raft_consensus.cc:493] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:06:41.238451 11789 raft_consensus.cc:515] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.240186 11789 leader_election.cc:290] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.240688 11584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" is_pre_election: true
I20260202 15:06:41.241218 11584 raft_consensus.cc:2468] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 0.
I20260202 15:06:41.241510 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9" is_pre_election: true
I20260202 15:06:41.242081 11735 raft_consensus.cc:2468] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 0.
I20260202 15:06:41.242041 11626 leader_election.cc:304] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:41.242848 11789 raft_consensus.cc:2804] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:06:41.243204 11789 raft_consensus.cc:493] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:06:41.243537 11789 raft_consensus.cc:3060] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:41.250180 11789 raft_consensus.cc:515] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.252220 11789 leader_election.cc:290] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.252460 11584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:41.252763 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:41.252996 11584 raft_consensus.cc:3060] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:41.253269 11735 raft_consensus.cc:3060] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:41.259645 11584 raft_consensus.cc:2468] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 1.
I20260202 15:06:41.259645 11735 raft_consensus.cc:2468] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 1.
I20260202 15:06:41.260830 11626 leader_election.cc:304] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:41.261476 11789 raft_consensus.cc:2804] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:41.261961 11789 raft_consensus.cc:697] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:41.262689 11789 consensus_queue.cc:237] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.270539 11482 catalog_manager.cc:5654] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:41.316975 11822 consensus_queue.cc:1048] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.327281 11824 consensus_queue.cc:1048] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:06:41.368248 11789 consensus_queue.cc:1048] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.380385 11842 consensus_queue.cc:1048] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:06:41.388968 11842 consensus_queue.cc:1048] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.400983 11786 consensus_queue.cc:1048] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.408048 11785 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.410444 11785 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.426645 11785 consensus_queue.cc:1048] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.443595 11785 consensus_queue.cc:1048] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.479341 11662 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:60600
I20260202 15:06:41.480080 11662 raft_consensus.cc:606] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Received request to transfer leadership to TS 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:41.485158 11661 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:60600
I20260202 15:06:41.485697 11661 raft_consensus.cc:606] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Received request to transfer leadership to TS 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:41.491401 11661 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:60600
I20260202 15:06:41.491964 11661 raft_consensus.cc:606] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Received request to transfer leadership to TS 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:41.496228 11661 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dd69d5f12dd2412089343e659c775418"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:60600
I20260202 15:06:41.496723 11661 raft_consensus.cc:606] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Received request to transfer leadership to TS 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:41.508970 11585 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
mode: GRACEFUL
new_leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:48040
I20260202 15:06:41.509364 11585 raft_consensus.cc:606] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Received request to transfer leadership to TS 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:41.522934 11735 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d"
dest_uuid: "1fcf043552924663b3127821864eb1e9"
mode: GRACEFUL
new_leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:45132
I20260202 15:06:41.523415 11735 raft_consensus.cc:606] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Received request to transfer leadership to TS 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:41.524235 11824 consensus_queue.cc:1048] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.529171 11735 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7"
dest_uuid: "1fcf043552924663b3127821864eb1e9"
mode: GRACEFUL
new_leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:45132
I20260202 15:06:41.529701 11735 raft_consensus.cc:606] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Received request to transfer leadership to TS 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:41.535629 11585 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "892cc90953b1478c864856ee708597c5"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
mode: GRACEFUL
new_leader_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:48040
I20260202 15:06:41.535754 11787 consensus_queue.cc:1048] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.536132 11585 raft_consensus.cc:606] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Received request to transfer leadership to TS 1fcf043552924663b3127821864eb1e9
I20260202 15:06:41.549933 11822 raft_consensus.cc:993] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5: : Instructing follower 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:41.550374 11822 raft_consensus.cc:1081] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Signalling peer 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:41.551858 11735 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "892cc90953b1478c864856ee708597c5"
dest_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:45116
I20260202 15:06:41.552476 11735 raft_consensus.cc:493] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:41.552847 11735 raft_consensus.cc:3060] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.557518 11735 raft_consensus.cc:515] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.561065 11735 leader_election.cc:290] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:41.561810 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892cc90953b1478c864856ee708597c5" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:41.562042 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892cc90953b1478c864856ee708597c5" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:41.562336 11585 raft_consensus.cc:3055] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:06:41.562538 11661 raft_consensus.cc:3060] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.562656 11585 raft_consensus.cc:740] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:41.563465 11585 consensus_queue.cc:260] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.564662 11585 raft_consensus.cc:3060] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.566892 11661 raft_consensus.cc:2468] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 2.
I20260202 15:06:41.567685 11702 leader_election.cc:304] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:41.568272 11854 raft_consensus.cc:2804] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:06:41.568646 11854 raft_consensus.cc:697] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 2 LEADER]: Becoming Leader. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:41.569247 11854 consensus_queue.cc:237] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.570372 11585 raft_consensus.cc:2468] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 2.
I20260202 15:06:41.576932 11483 catalog_manager.cc:5654] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 reported cstate change: term changed from 1 to 2, leader changed from 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193) to 1fcf043552924663b3127821864eb1e9 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1fcf043552924663b3127821864eb1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: HEALTHY } } }
I20260202 15:06:41.611850 11789 raft_consensus.cc:993] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:41.612280 11842 raft_consensus.cc:1081] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Signalling peer 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:41.613600 11585 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:48012
I20260202 15:06:41.614130 11585 raft_consensus.cc:493] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:41.614497 11585 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.620240 11585 raft_consensus.cc:515] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.622128 11585 leader_election.cc:290] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.622641 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:41.622641 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:41.623394 11735 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.623445 11661 raft_consensus.cc:3055] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:06:41.623875 11661 raft_consensus.cc:740] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:41.624635 11661 consensus_queue.cc:260] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.625761 11661 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.629423 11735 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:41.630398 11549 leader_election.cc:304] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:41.631114 11822 raft_consensus.cc:2804] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:06:41.631539 11822 raft_consensus.cc:697] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Becoming Leader. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:41.632112 11661 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:41.632230 11822 consensus_queue.cc:237] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.638398 11483 catalog_manager.cc:5654] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 reported cstate change: term changed from 1 to 2, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193). New cstate: current_term: 2 leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:41.646819 11842 consensus_queue.cc:1048] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.652479 11789 raft_consensus.cc:993] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:41.652872 11786 raft_consensus.cc:1081] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Signalling peer 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:41.654266 11586 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "dd69d5f12dd2412089343e659c775418"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:48012
I20260202 15:06:41.654774 11586 raft_consensus.cc:493] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:41.655139 11586 raft_consensus.cc:3060] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.658803 11786 consensus_queue.cc:1048] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.666514 11586 raft_consensus.cc:515] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.668917 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:41.670244 11585 raft_consensus.cc:1240] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Rejecting Update request from peer 9079295b8e5e40dfbc4d95468b0a0f71 for earlier term 1. Current term is 2. Ops: []
I20260202 15:06:41.670647 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:41.669864 11586 leader_election.cc:290] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.671147 11734 raft_consensus.cc:3060] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.672273 11789 consensus_queue.cc:1059] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:06:41.669492 11661 raft_consensus.cc:3055] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:06:41.673416 11661 raft_consensus.cc:740] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:41.674165 11661 consensus_queue.cc:260] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.675330 11661 raft_consensus.cc:3060] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.677156 11734 raft_consensus.cc:2468] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:41.679107 11796 raft_consensus.cc:993] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:41.679472 11796 raft_consensus.cc:1081] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Signalling peer 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:41.680306 11549 leader_election.cc:304] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:41.680893 11822 raft_consensus.cc:2804] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:06:41.681277 11661 raft_consensus.cc:2468] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:41.681314 11822 raft_consensus.cc:697] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Becoming Leader. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
W20260202 15:06:41.681805 11789 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260202 15:06:41.682193 11822 consensus_queue.cc:237] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.681633 11585 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:48012
I20260202 15:06:41.684545 11585 raft_consensus.cc:493] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:41.684929 11585 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.689291 11479 catalog_manager.cc:5654] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 reported cstate change: term changed from 1 to 2, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193). New cstate: current_term: 2 leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:41.692245 11585 raft_consensus.cc:515] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:41.694249 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:41.694516 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:41.695061 11734 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.695410 11661 raft_consensus.cc:3055] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:06:41.695744 11661 raft_consensus.cc:740] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:41.696475 11661 consensus_queue.cc:260] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:41.697469 11661 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.698661 11585 leader_election.cc:290] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.700629 11734 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:41.701520 11549 leader_election.cc:304] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:41.702204 11822 raft_consensus.cc:2804] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:06:41.702613 11822 raft_consensus.cc:697] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Becoming Leader. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:41.703027 11661 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:41.703353 11822 consensus_queue.cc:237] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:41.709262 11479 catalog_manager.cc:5654] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 reported cstate change: term changed from 1 to 2, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193). New cstate: current_term: 2 leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: HEALTHY } } }
I20260202 15:06:41.804047 11796 raft_consensus.cc:993] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:41.804540 11789 raft_consensus.cc:1081] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Signalling peer 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:41.805939 11585 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:48012
I20260202 15:06:41.806417 11585 raft_consensus.cc:493] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:41.806746 11585 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.811774 11585 raft_consensus.cc:515] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.814783 11585 leader_election.cc:290] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.815248 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:41.815665 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:41.815977 11661 raft_consensus.cc:3055] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:06:41.816356 11661 raft_consensus.cc:740] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:41.816437 11734 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.817054 11661 consensus_queue.cc:260] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.817863 11661 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.821659 11661 raft_consensus.cc:2468] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:41.822583 11734 raft_consensus.cc:2468] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:41.822623 11552 leader_election.cc:304] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:41.823531 11822 raft_consensus.cc:2804] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:06:41.823966 11822 raft_consensus.cc:697] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Becoming Leader. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:41.824752 11822 consensus_queue.cc:237] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.831281 11479 catalog_manager.cc:5654] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 reported cstate change: term changed from 1 to 2, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193). New cstate: current_term: 2 leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:41.848495 11785 raft_consensus.cc:993] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9: : Instructing follower 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:41.848865 11854 raft_consensus.cc:1081] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Signalling peer 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:41.850172 11661 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:60594
I20260202 15:06:41.850620 11661 raft_consensus.cc:493] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:41.850997 11661 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.856954 11661 raft_consensus.cc:515] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.858471 11661 leader_election.cc:290] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 2 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.859356 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:41.859418 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:41.859898 11585 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.859978 11734 raft_consensus.cc:3055] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:06:41.860309 11734 raft_consensus.cc:740] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:41.860963 11734 consensus_queue.cc:260] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.862056 11734 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.864382 11585 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 2.
I20260202 15:06:41.865352 11626 leader_election.cc:304] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:41.865988 11789 raft_consensus.cc:2804] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:06:41.866382 11789 raft_consensus.cc:697] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:41.867123 11789 consensus_queue.cc:237] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.867949 11734 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 2.
I20260202 15:06:41.870965 11822 raft_consensus.cc:993] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5: : Instructing follower 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:41.871311 11822 raft_consensus.cc:1081] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Signalling peer 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:41.872830 11662 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:60596
I20260202 15:06:41.873325 11662 raft_consensus.cc:493] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:41.873652 11662 raft_consensus.cc:3060] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.875236 11483 catalog_manager.cc:5654] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 1 to 2, leader changed from 1fcf043552924663b3127821864eb1e9 (127.11.46.195) to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:41.879779 11662 raft_consensus.cc:515] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.881878 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:41.882115 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21c950e2b059499a87376daacd8a7c14" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:41.882494 11585 raft_consensus.cc:3055] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:06:41.882786 11662 leader_election.cc:290] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 2 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:41.882918 11585 raft_consensus.cc:740] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:41.882591 11734 raft_consensus.cc:3060] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.883852 11585 consensus_queue.cc:260] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.885078 11585 raft_consensus.cc:3060] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:41.889693 11734 raft_consensus.cc:2468] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 2.
I20260202 15:06:41.890635 11624 leader_election.cc:304] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:41.891191 11585 raft_consensus.cc:2468] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 2.
I20260202 15:06:41.891397 11789 raft_consensus.cc:2804] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:06:41.891835 11789 raft_consensus.cc:697] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:41.892536 11789 consensus_queue.cc:237] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:41.898613 11483 catalog_manager.cc:5654] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 1 to 2, leader changed from 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193) to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:42.020418 11662 raft_consensus.cc:1275] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.021446 11854 consensus_queue.cc:1048] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.027822 11864 raft_consensus.cc:993] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9: : Instructing follower 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:42.029225 11864 raft_consensus.cc:1081] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Signalling peer 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:42.031009 11662 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:60594
I20260202 15:06:42.031526 11662 raft_consensus.cc:493] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:42.032613 11585 raft_consensus.cc:1275] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.031895 11662 raft_consensus.cc:3060] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:42.034078 11865 consensus_queue.cc:1048] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:06:42.041548 11662 raft_consensus.cc:515] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:42.043133 11662 leader_election.cc:290] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 2 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:42.043700 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:42.044314 11585 raft_consensus.cc:3060] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:42.044708 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:42.045321 11734 raft_consensus.cc:3055] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:06:42.045665 11734 raft_consensus.cc:740] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:42.046520 11734 consensus_queue.cc:260] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:42.047796 11734 raft_consensus.cc:3060] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:42.049881 11585 raft_consensus.cc:2468] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 2.
I20260202 15:06:42.050971 11626 leader_election.cc:304] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:42.051945 11789 raft_consensus.cc:2804] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:06:42.052386 11789 raft_consensus.cc:697] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:42.053123 11734 raft_consensus.cc:2468] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 2.
I20260202 15:06:42.053043 11789 consensus_queue.cc:237] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:42.058465 11483 catalog_manager.cc:5654] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 1 to 2, leader changed from 1fcf043552924663b3127821864eb1e9 (127.11.46.195) to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:42.117655 11734 raft_consensus.cc:1275] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.118742 11822 consensus_queue.cc:1048] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.127977 11662 raft_consensus.cc:1275] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.129056 11868 consensus_queue.cc:1048] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.168532 11734 raft_consensus.cc:1275] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.169461 11868 consensus_queue.cc:1048] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.175839 11662 raft_consensus.cc:1275] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.176867 11824 consensus_queue.cc:1048] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.257225 11662 raft_consensus.cc:1275] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.258167 11824 consensus_queue.cc:1048] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.264815 11734 raft_consensus.cc:1275] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.266253 11824 consensus_queue.cc:1048] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.293190 11734 raft_consensus.cc:1275] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.294270 11789 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.301023 11585 raft_consensus.cc:1275] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.302632 11786 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.383708 11734 raft_consensus.cc:1275] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.384842 11824 consensus_queue.cc:1048] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.391489 11662 raft_consensus.cc:1275] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.392716 11868 consensus_queue.cc:1048] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.418700 11585 raft_consensus.cc:1275] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.419783 11786 consensus_queue.cc:1048] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.426939 11734 raft_consensus.cc:1275] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.428042 11796 consensus_queue.cc:1048] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.448113 11734 raft_consensus.cc:1275] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.448930 11796 consensus_queue.cc:1048] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:42.455806 11585 raft_consensus.cc:1275] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:42.457223 11786 consensus_queue.cc:1048] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:44.562903 11451 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20260202 15:06:44.563617 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:44.582350 11585 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
mode: GRACEFUL
new_leader_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:48046
I20260202 15:06:44.582876 11585 raft_consensus.cc:606] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Received request to transfer leadership to TS 1fcf043552924663b3127821864eb1e9
I20260202 15:06:44.591646 11662 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:60614
I20260202 15:06:44.592089 11662 raft_consensus.cc:606] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Received request to transfer leadership to TS 1fcf043552924663b3127821864eb1e9
I20260202 15:06:44.593142 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260202 15:06:44.593561 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:44.684422 11824 raft_consensus.cc:993] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5: : Instructing follower 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:44.684885 11881 raft_consensus.cc:1081] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Signalling peer 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:44.686221 11735 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e"
dest_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:45116
I20260202 15:06:44.686760 11735 raft_consensus.cc:493] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:44.687173 11735 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:44.693403 11735 raft_consensus.cc:515] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:44.695204 11735 leader_election.cc:290] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:44.696059 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:44.696231 11662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:44.696575 11585 raft_consensus.cc:3055] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:06:44.696828 11662 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:44.696949 11585 raft_consensus.cc:740] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:44.698064 11585 consensus_queue.cc:260] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:44.698997 11585 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:44.701601 11662 raft_consensus.cc:2468] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 3.
I20260202 15:06:44.702633 11702 leader_election.cc:304] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:44.703392 11876 raft_consensus.cc:2804] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:06:44.703429 11585 raft_consensus.cc:2468] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 3.
I20260202 15:06:44.704663 11876 raft_consensus.cc:697] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Becoming Leader. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:44.705403 11876 consensus_queue.cc:237] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:44.711964 11482 catalog_manager.cc:5654] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 reported cstate change: term changed from 2 to 3, leader changed from 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193) to 1fcf043552924663b3127821864eb1e9 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1fcf043552924663b3127821864eb1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: HEALTHY } } }
I20260202 15:06:44.883483 11877 raft_consensus.cc:993] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:44.883910 11877 raft_consensus.cc:1081] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Signalling peer 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:44.885192 11735 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d"
dest_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:45112
I20260202 15:06:44.885588 11735 raft_consensus.cc:493] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:44.885820 11735 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:44.889846 11735 raft_consensus.cc:515] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:44.891144 11735 leader_election.cc:290] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:44.891971 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:44.892446 11662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:44.892588 11585 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:44.893093 11662 raft_consensus.cc:3055] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:06:44.893436 11662 raft_consensus.cc:740] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:44.894168 11662 consensus_queue.cc:260] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:44.895354 11662 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:44.898523 11585 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 3.
I20260202 15:06:44.899452 11701 leader_election.cc:304] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:44.900162 11876 raft_consensus.cc:2804] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:06:44.900593 11876 raft_consensus.cc:697] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Becoming Leader. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:44.900885 11662 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 3.
I20260202 15:06:44.901314 11876 consensus_queue.cc:237] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:44.909081 11482 catalog_manager.cc:5654] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 reported cstate change: term changed from 2 to 3, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 1fcf043552924663b3127821864eb1e9 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1fcf043552924663b3127821864eb1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: HEALTHY } } }
I20260202 15:06:45.089071 11585 raft_consensus.cc:1275] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:45.090183 11876 consensus_queue.cc:1048] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:45.099691 11662 raft_consensus.cc:1275] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:45.103978 11876 consensus_queue.cc:1048] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:06:45.348654 11662 raft_consensus.cc:1275] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:45.349819 11882 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:45.357666 11585 raft_consensus.cc:1275] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:45.359671 11882 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:45.594444 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:45.601020 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:45.601353 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:46.652065 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:46.659986 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:46.660459 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260202 15:06:46.914458 11514 debug-util.cc:398] Leaking SignalData structure 0x7b0800061060 after lost signal to thread 11454
W20260202 15:06:46.915323 11514 debug-util.cc:398] Leaking SignalData structure 0x7b0800061740 after lost signal to thread 11517
W20260202 15:06:46.916163 11514 debug-util.cc:398] Leaking SignalData structure 0x7b08001d2e00 after lost signal to thread 11610
W20260202 15:06:46.916913 11514 debug-util.cc:398] Leaking SignalData structure 0x7b0800159880 after lost signal to thread 11686
W20260202 15:06:46.917644 11514 debug-util.cc:398] Leaking SignalData structure 0x7b08001a3720 after lost signal to thread 11760
I20260202 15:06:47.661317 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:47.666414 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:47.666821 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:48.667824 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:48.672539 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:48.672932 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:49.674096 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:49.679037 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:49.679473 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:50.737983 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:50.743160 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:50.743533 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260202 15:06:51.355388 11757 debug-util.cc:398] Leaking SignalData structure 0x7b080016e060 after lost signal to thread 11454
W20260202 15:06:51.356142 11757 debug-util.cc:398] Leaking SignalData structure 0x7b080016e740 after lost signal to thread 11517
W20260202 15:06:51.356923 11757 debug-util.cc:398] Leaking SignalData structure 0x7b08001d3000 after lost signal to thread 11610
W20260202 15:06:51.357671 11757 debug-util.cc:398] Leaking SignalData structure 0x7b0800061b20 after lost signal to thread 11686
W20260202 15:06:51.358359 11757 debug-util.cc:398] Leaking SignalData structure 0x7b08001e17a0 after lost signal to thread 11760
I20260202 15:06:51.744582 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:51.753609 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:51.754338 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:52.756543 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:52.766923 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:52.768380 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:53.769433 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:53.773423 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:53.773681 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:54.795107 11451 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20260202 15:06:54.798954 11586 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
mode: GRACEFUL
new_leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:48040
I20260202 15:06:54.799713 11586 raft_consensus.cc:606] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Received request to transfer leadership to TS 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:54.803926 11734 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e"
dest_uuid: "1fcf043552924663b3127821864eb1e9"
mode: GRACEFUL
new_leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:45132
I20260202 15:06:54.804239 11734 raft_consensus.cc:606] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Received request to transfer leadership to TS 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:54.807765 11586 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
mode: GRACEFUL
new_leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:48040
I20260202 15:06:54.808089 11586 raft_consensus.cc:606] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Received request to transfer leadership to TS 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:54.811471 11913 raft_consensus.cc:993] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9: : Instructing follower 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:54.811661 11586 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dd69d5f12dd2412089343e659c775418"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
mode: GRACEFUL
new_leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:48040
I20260202 15:06:54.811966 11911 raft_consensus.cc:1081] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Signalling peer 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:54.812098 11586 raft_consensus.cc:606] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Received request to transfer leadership to TS 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:54.813364 11662 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:60594
I20260202 15:06:54.813930 11662 raft_consensus.cc:493] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:54.814209 11662 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:54.818984 11662 raft_consensus.cc:515] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:54.820576 11662 leader_election.cc:290] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 4 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:54.821038 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:54.821141 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e50d403143c44cbb571fb0fae1a0d4e" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:54.821592 11735 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d"
dest_uuid: "1fcf043552924663b3127821864eb1e9"
mode: GRACEFUL
new_leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:45132
I20260202 15:06:54.821713 11586 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:54.821753 11734 raft_consensus.cc:3055] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:06:54.822043 11735 raft_consensus.cc:606] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Received request to transfer leadership to TS 9079295b8e5e40dfbc4d95468b0a0f71
I20260202 15:06:54.822110 11734 raft_consensus.cc:740] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:54.822870 11734 consensus_queue.cc:260] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:54.823858 11734 raft_consensus.cc:3060] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:54.827342 11586 raft_consensus.cc:2468] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 4.
I20260202 15:06:54.828541 11734 raft_consensus.cc:2468] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 4.
I20260202 15:06:54.829398 11624 leader_election.cc:304] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:54.830073 11904 raft_consensus.cc:2804] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:06:54.830832 11904 raft_consensus.cc:697] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:54.831521 11904 consensus_queue.cc:237] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:54.836361 11916 raft_consensus.cc:993] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5: : Instructing follower 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:54.836779 11912 raft_consensus.cc:1081] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Signalling peer 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:54.837848 11662 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:60596
I20260202 15:06:54.838258 11662 raft_consensus.cc:493] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:54.838531 11662 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:54.841445 11479 catalog_manager.cc:5654] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 3 to 4, leader changed from 1fcf043552924663b3127821864eb1e9 (127.11.46.195) to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 4 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:54.843358 11662 raft_consensus.cc:515] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:54.845499 11662 leader_election.cc:290] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 3 election: Requested vote from peers 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527), 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249)
I20260202 15:06:54.845531 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:54.846129 11734 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:54.847293 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:54.847672 11586 raft_consensus.cc:3055] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:06:54.847922 11586 raft_consensus.cc:740] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:54.848749 11586 consensus_queue.cc:260] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:54.850076 11586 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:54.850569 11734 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 3.
I20260202 15:06:54.851505 11624 leader_election.cc:304] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:54.852257 11904 raft_consensus.cc:2804] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:06:54.852636 11904 raft_consensus.cc:697] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:54.853526 11904 consensus_queue.cc:237] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:54.855767 11586 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 3.
I20260202 15:06:54.862238 11479 catalog_manager.cc:5654] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 2 to 3, leader changed from 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193) to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 3 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:55.081400 11916 raft_consensus.cc:993] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5: : Instructing follower 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:55.081881 11912 raft_consensus.cc:1081] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Signalling peer 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:55.082716 11913 raft_consensus.cc:993] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9: : Instructing follower 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:55.083212 11911 raft_consensus.cc:1081] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Signalling peer 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:55.083417 11662 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:60596
I20260202 15:06:55.083853 11662 raft_consensus.cc:493] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:55.084117 11662 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:55.084542 11661 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:60594
I20260202 15:06:55.084962 11661 raft_consensus.cc:493] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:55.085306 11661 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:55.090092 11662 raft_consensus.cc:515] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:55.091141 11661 raft_consensus.cc:515] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:55.091969 11662 leader_election.cc:290] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 3 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:55.092487 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:55.092957 11734 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:55.092994 11661 leader_election.cc:290] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 4 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:55.093401 11585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:55.093533 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:55.093796 11585 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:55.094004 11735 raft_consensus.cc:3055] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:06:55.094290 11735 raft_consensus.cc:740] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:55.094973 11735 consensus_queue.cc:260] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:55.096016 11735 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:55.097584 11585 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 4.
I20260202 15:06:55.098551 11626 leader_election.cc:304] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1008aec55e4adf85c21ea22e0ee0c5, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:55.099080 11734 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 3.
I20260202 15:06:55.092767 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:55.099884 11904 raft_consensus.cc:2804] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:06:55.099983 11624 leader_election.cc:304] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:55.100458 11904 raft_consensus.cc:697] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:55.100816 11914 raft_consensus.cc:2804] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:06:55.101372 11904 consensus_queue.cc:237] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:55.101377 11914 raft_consensus.cc:697] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:55.101977 11735 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 4.
I20260202 15:06:55.102631 11914 consensus_queue.cc:237] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:55.100358 11586 raft_consensus.cc:3055] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:06:55.104226 11586 raft_consensus.cc:740] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:55.104982 11586 consensus_queue.cc:260] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:55.106401 11586 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:55.108327 11479 catalog_manager.cc:5654] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 3 to 4, leader changed from 1fcf043552924663b3127821864eb1e9 (127.11.46.195) to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 4 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:55.111325 11586 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 3.
I20260202 15:06:55.117652 11479 catalog_manager.cc:5654] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 2 to 3, leader changed from 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193) to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 3 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:55.184837 11916 raft_consensus.cc:993] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5: : Instructing follower 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:55.185261 11912 raft_consensus.cc:1081] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Signalling peer 9079295b8e5e40dfbc4d95468b0a0f71 to start an election
I20260202 15:06:55.186733 11661 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "dd69d5f12dd2412089343e659c775418"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
from {username='slave'} at 127.0.0.1:60596
I20260202 15:06:55.187203 11661 raft_consensus.cc:493] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:55.187481 11661 raft_consensus.cc:3060] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:55.191636 11661 raft_consensus.cc:515] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:55.192910 11661 leader_election.cc:290] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 3 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:55.193903 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:55.193939 11734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd69d5f12dd2412089343e659c775418" candidate_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:55.194458 11734 raft_consensus.cc:3060] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:55.194449 11586 raft_consensus.cc:3055] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:06:55.195010 11586 raft_consensus.cc:740] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:55.195680 11586 consensus_queue.cc:260] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:55.196647 11586 raft_consensus.cc:3060] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:55.199916 11734 raft_consensus.cc:2468] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 3.
I20260202 15:06:55.201104 11586 raft_consensus.cc:2468] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9079295b8e5e40dfbc4d95468b0a0f71 in term 3.
I20260202 15:06:55.201584 11624 leader_election.cc:304] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 9079295b8e5e40dfbc4d95468b0a0f71; no voters:
I20260202 15:06:55.202430 11914 raft_consensus.cc:2804] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:06:55.202894 11914 raft_consensus.cc:697] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Becoming Leader. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:55.203693 11914 consensus_queue.cc:237] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:55.211071 11479 catalog_manager.cc:5654] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 reported cstate change: term changed from 2 to 3, leader changed from 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193) to 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194). New cstate: current_term: 3 leader_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:55.240710 11734 raft_consensus.cc:1275] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:55.242151 11914 consensus_queue.cc:1048] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:55.253682 11586 raft_consensus.cc:1275] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:55.256702 11904 consensus_queue.cc:1048] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:55.334273 11734 raft_consensus.cc:1275] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:06:55.335206 11904 consensus_queue.cc:1048] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:06:55.342154 11586 raft_consensus.cc:1275] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:06:55.343751 11922 consensus_queue.cc:1048] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:06:55.588636 11586 raft_consensus.cc:1275] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:06:55.589515 11922 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:06:55.596844 11734 raft_consensus.cc:1275] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:06:55.598153 11904 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:06:55.625288 11734 raft_consensus.cc:1275] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:55.626349 11904 consensus_queue.cc:1048] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:55.633581 11586 raft_consensus.cc:1275] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:55.635587 11914 consensus_queue.cc:1048] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:55.732626 11734 raft_consensus.cc:1275] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:55.733893 11922 consensus_queue.cc:1048] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:55.745023 11586 raft_consensus.cc:1275] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Refusing update from remote peer 9079295b8e5e40dfbc4d95468b0a0f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:55.746259 11922 consensus_queue.cc:1048] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:57.882869 11451 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20260202 15:06:57.883672 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:57.891381 11661 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:60614
I20260202 15:06:57.892213 11661 raft_consensus.cc:606] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Received request to transfer leadership to TS 1fcf043552924663b3127821864eb1e9
I20260202 15:06:57.895426 11661 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:60614
I20260202 15:06:57.895987 11661 raft_consensus.cc:606] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Received request to transfer leadership to TS 1fcf043552924663b3127821864eb1e9
I20260202 15:06:57.898478 11661 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:60614
I20260202 15:06:57.899185 11661 raft_consensus.cc:606] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 LEADER]: Received request to transfer leadership to TS 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:57.902235 11661 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:60614
I20260202 15:06:57.903173 11661 raft_consensus.cc:606] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Received request to transfer leadership to TS 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:57.906116 11661 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fdf8b5e061f54d348072cddc0fbc2cda"
dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
mode: GRACEFUL
new_leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:60614
I20260202 15:06:57.907397 11661 raft_consensus.cc:606] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Received request to transfer leadership to TS 4d1008aec55e4adf85c21ea22e0ee0c5
I20260202 15:06:57.908584 11941 raft_consensus.cc:993] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:57.908943 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260202 15:06:57.909452 11939 raft_consensus.cc:1081] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Signalling peer 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:57.909727 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:57.910478 11939 raft_consensus.cc:993] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:57.910967 11939 raft_consensus.cc:1081] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Signalling peer 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:57.911974 11734 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb"
dest_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:45112
I20260202 15:06:57.912714 11734 raft_consensus.cc:493] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:57.913199 11734 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:57.913697 11585 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:48012
I20260202 15:06:57.914626 11585 raft_consensus.cc:493] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:57.915102 11585 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:57.921286 11734 raft_consensus.cc:515] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:57.923426 11585 raft_consensus.cc:515] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:57.924889 11661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:57.925546 11661 raft_consensus.cc:3055] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:06:57.925438 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a70ec9fde0b4e8a896aed629584f8cb" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:57.925953 11661 raft_consensus.cc:740] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:57.926298 11586 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:57.923767 11734 leader_election.cc:290] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 4 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249)
I20260202 15:06:57.927389 11585 leader_election.cc:290] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 4 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:57.926995 11661 consensus_queue.cc:260] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:57.927822 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:57.928421 11661 raft_consensus.cc:3060] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:57.928620 11735 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:57.932453 11586 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 4.
I20260202 15:06:57.933676 11701 leader_election.cc:304] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:57.934326 11662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b4a87d39cb14268991cb083677738d2" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:57.934887 11662 raft_consensus.cc:3055] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:06:57.934820 11911 raft_consensus.cc:2804] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:06:57.935252 11662 raft_consensus.cc:740] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:57.935526 11661 raft_consensus.cc:2468] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 4.
I20260202 15:06:57.936446 11735 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 4.
I20260202 15:06:57.936287 11662 consensus_queue.cc:260] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:57.936861 11911 raft_consensus.cc:697] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 4 LEADER]: Becoming Leader. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:57.938030 11662 raft_consensus.cc:3060] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:06:57.938074 11911 consensus_queue.cc:237] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } }
I20260202 15:06:57.942058 11549 leader_election.cc:304] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:57.945286 11662 raft_consensus.cc:2468] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 4.
I20260202 15:06:57.947065 11946 raft_consensus.cc:2804] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:06:57.948374 11946 raft_consensus.cc:697] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 LEADER]: Becoming Leader. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:57.948294 11482 catalog_manager.cc:5654] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 reported cstate change: term changed from 3 to 4, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 1fcf043552924663b3127821864eb1e9 (127.11.46.195). New cstate: current_term: 4 leader_uuid: "1fcf043552924663b3127821864eb1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:57.949326 11946 consensus_queue.cc:237] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:57.957965 11479 catalog_manager.cc:5654] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 reported cstate change: term changed from 3 to 4, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193). New cstate: current_term: 4 leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:58.063591 11943 raft_consensus.cc:993] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:58.063934 11941 raft_consensus.cc:1081] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Signalling peer 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:58.065120 11586 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fdf8b5e061f54d348072cddc0fbc2cda"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:48012
I20260202 15:06:58.065532 11586 raft_consensus.cc:493] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:58.065796 11586 raft_consensus.cc:3060] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:58.069494 11586 raft_consensus.cc:515] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:58.070688 11586 leader_election.cc:290] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:58.071381 11662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdf8b5e061f54d348072cddc0fbc2cda" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:58.071601 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdf8b5e061f54d348072cddc0fbc2cda" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:58.071843 11662 raft_consensus.cc:3055] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:06:58.072098 11735 raft_consensus.cc:3060] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:58.072180 11662 raft_consensus.cc:740] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:58.072880 11662 consensus_queue.cc:260] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:58.074074 11662 raft_consensus.cc:3060] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:06:58.077780 11735 raft_consensus.cc:2468] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:58.078896 11549 leader_election.cc:304] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:58.079514 11662 raft_consensus.cc:2468] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 2.
I20260202 15:06:58.079522 11946 raft_consensus.cc:2804] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:06:58.080037 11946 raft_consensus.cc:697] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Becoming Leader. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:58.080724 11946 consensus_queue.cc:237] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:58.086956 11479 catalog_manager.cc:5654] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 reported cstate change: term changed from 1 to 2, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193). New cstate: current_term: 2 leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:58.094812 11943 raft_consensus.cc:993] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:58.095142 11941 raft_consensus.cc:1081] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 LEADER]: Signalling peer 4d1008aec55e4adf85c21ea22e0ee0c5 to start an election
I20260202 15:06:58.096292 11586 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d"
dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
from {username='slave'} at 127.0.0.1:48012
I20260202 15:06:58.096671 11586 raft_consensus.cc:493] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:58.096917 11586 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:06:58.100772 11586 raft_consensus.cc:515] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:58.102002 11586 leader_election.cc:290] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 5 election: Requested vote from peers 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459), 1fcf043552924663b3127821864eb1e9 (127.11.46.195:44527)
I20260202 15:06:58.102861 11735 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "1fcf043552924663b3127821864eb1e9"
I20260202 15:06:58.102861 11662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f30fdf5d624adda37324efe532a47d" candidate_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:58.103606 11735 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:06:58.103607 11662 raft_consensus.cc:3055] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:06:58.104096 11662 raft_consensus.cc:740] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:58.104852 11662 consensus_queue.cc:260] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:58.106052 11662 raft_consensus.cc:3060] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:06:58.109635 11735 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 5.
I20260202 15:06:58.110541 11549 leader_election.cc:304] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:58.111409 11946 raft_consensus.cc:2804] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:06:58.111532 11662 raft_consensus.cc:2468] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1008aec55e4adf85c21ea22e0ee0c5 in term 5.
I20260202 15:06:58.111815 11946 raft_consensus.cc:697] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 5 LEADER]: Becoming Leader. State: Replica: 4d1008aec55e4adf85c21ea22e0ee0c5, State: Running, Role: LEADER
I20260202 15:06:58.112725 11946 consensus_queue.cc:237] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:58.119218 11479 catalog_manager.cc:5654] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 reported cstate change: term changed from 4 to 5, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193). New cstate: current_term: 5 leader_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260202 15:06:58.150442 11943 raft_consensus.cc:993] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71: : Instructing follower 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:58.150894 11941 raft_consensus.cc:1081] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Signalling peer 1fcf043552924663b3127821864eb1e9 to start an election
I20260202 15:06:58.152114 11735 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7"
dest_uuid: "1fcf043552924663b3127821864eb1e9"
from {username='slave'} at 127.0.0.1:45112
I20260202 15:06:58.152554 11735 raft_consensus.cc:493] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:06:58.152827 11735 raft_consensus.cc:3060] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:58.157253 11735 raft_consensus.cc:515] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:58.158766 11735 leader_election.cc:290] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249), 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194:46459)
I20260202 15:06:58.159512 11586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5"
I20260202 15:06:58.159845 11662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f196b3798944459a69108f9d93b0d7" candidate_uuid: "1fcf043552924663b3127821864eb1e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9079295b8e5e40dfbc4d95468b0a0f71"
I20260202 15:06:58.160179 11586 raft_consensus.cc:3060] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:58.160372 11662 raft_consensus.cc:3055] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:06:58.160665 11662 raft_consensus.cc:740] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9079295b8e5e40dfbc4d95468b0a0f71, State: Running, Role: LEADER
I20260202 15:06:58.161365 11662 consensus_queue.cc:260] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:58.162608 11662 raft_consensus.cc:3060] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:06:58.166172 11586 raft_consensus.cc:2468] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 3.
I20260202 15:06:58.167097 11701 leader_election.cc:304] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf043552924663b3127821864eb1e9, 4d1008aec55e4adf85c21ea22e0ee0c5; no voters:
I20260202 15:06:58.167747 11911 raft_consensus.cc:2804] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:06:58.168119 11911 raft_consensus.cc:697] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Becoming Leader. State: Replica: 1fcf043552924663b3127821864eb1e9, State: Running, Role: LEADER
I20260202 15:06:58.168196 11662 raft_consensus.cc:2468] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcf043552924663b3127821864eb1e9 in term 3.
I20260202 15:06:58.168999 11911 consensus_queue.cc:237] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } }
I20260202 15:06:58.175089 11479 catalog_manager.cc:5654] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 reported cstate change: term changed from 2 to 3, leader changed from 9079295b8e5e40dfbc4d95468b0a0f71 (127.11.46.194) to 1fcf043552924663b3127821864eb1e9 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1fcf043552924663b3127821864eb1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 } health_report { overall_health: HEALTHY } } }
I20260202 15:06:58.371603 11586 raft_consensus.cc:1275] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:06:58.372570 11911 consensus_queue.cc:1048] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:06:58.380437 11662 raft_consensus.cc:1275] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:06:58.381527 11911 consensus_queue.cc:1048] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:06:58.385674 11735 raft_consensus.cc:1275] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:06:58.387187 11946 consensus_queue.cc:1048] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:06:58.397823 11662 raft_consensus.cc:1275] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:06:58.398834 11947 consensus_queue.cc:1048] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:06:58.578058 11662 raft_consensus.cc:1275] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:58.578877 11959 consensus_queue.cc:1048] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:58.586678 11586 raft_consensus.cc:1275] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Refusing update from remote peer 1fcf043552924663b3127821864eb1e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:06:58.587764 11959 consensus_queue.cc:1048] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1008aec55e4adf85c21ea22e0ee0c5" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:06:58.618248 11735 raft_consensus.cc:1275] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 5 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:06:58.619088 11947 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:06:58.626034 11662 raft_consensus.cc:1275] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 5 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:06:58.627933 11967 consensus_queue.cc:1048] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:06:58.640363 11662 raft_consensus.cc:1275] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:58.641129 11967 consensus_queue.cc:1048] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9079295b8e5e40dfbc4d95468b0a0f71" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:58.647598 11735 raft_consensus.cc:1275] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Refusing update from remote peer 4d1008aec55e4adf85c21ea22e0ee0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:06:58.648949 11967 consensus_queue.cc:1048] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf043552924663b3127821864eb1e9" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:06:58.911357 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:58.915367 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:58.915715 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:06:59.916957 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:06:59.922303 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:06:59.922979 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:00.923935 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:00.928166 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:00.928488 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:01.929536 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:01.934602 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:01.935006 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:02.936045 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:02.940966 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:02.941404 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:03.942368 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:03.946893 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:03.947352 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:04.948344 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:04.953255 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:04.953594 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:05.954437 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:05.959131 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:05.959455 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:06.960377 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:06.965703 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:06.966135 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:07.992551 11451 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20260202 15:07:07.993492 11451 tablet_server.cc:178] TabletServer@127.11.46.193:0 shutting down...
W20260202 15:07:08.012199 11626 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260202 15:07:08.016670 11626 consensus_peers.cc:597] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 -> Peer 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249): Couldn't send request to peer 4d1008aec55e4adf85c21ea22e0ee0c5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:08.021168 11451 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260202 15:07:08.022010 11451 tablet_replica.cc:333] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5: stopping tablet replica
I20260202 15:07:08.023074 11451 raft_consensus.cc:2243] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.023774 11451 raft_consensus.cc:2272] T 5e50d403143c44cbb571fb0fae1a0d4e P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.026001 11451 tablet_replica.cc:333] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5: stopping tablet replica
I20260202 15:07:08.026554 11451 raft_consensus.cc:2243] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.027065 11451 raft_consensus.cc:2272] T 28f196b3798944459a69108f9d93b0d7 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.028859 11451 tablet_replica.cc:333] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5: stopping tablet replica
I20260202 15:07:08.029397 11451 raft_consensus.cc:2243] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.029911 11451 raft_consensus.cc:2272] T 892cc90953b1478c864856ee708597c5 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.031749 11451 tablet_replica.cc:333] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5: stopping tablet replica
I20260202 15:07:08.032294 11451 raft_consensus.cc:2243] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 5 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.033461 11451 raft_consensus.cc:2272] T 90f30fdf5d624adda37324efe532a47d P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.035259 11451 tablet_replica.cc:333] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5: stopping tablet replica
I20260202 15:07:08.035744 11451 raft_consensus.cc:2243] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.036232 11451 raft_consensus.cc:2272] T 21c950e2b059499a87376daacd8a7c14 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.037802 11451 tablet_replica.cc:333] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5: stopping tablet replica
I20260202 15:07:08.038300 11451 raft_consensus.cc:2243] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.039595 11451 raft_consensus.cc:2272] T 9b4a87d39cb14268991cb083677738d2 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.041464 11451 tablet_replica.cc:333] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5: stopping tablet replica
I20260202 15:07:08.041970 11451 raft_consensus.cc:2243] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.042443 11451 raft_consensus.cc:2272] T dd69d5f12dd2412089343e659c775418 P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.044247 11451 tablet_replica.cc:333] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5: stopping tablet replica
I20260202 15:07:08.044734 11451 raft_consensus.cc:2243] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.045747 11451 raft_consensus.cc:2272] T fdf8b5e061f54d348072cddc0fbc2cda P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.048622 11451 tablet_replica.cc:333] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5: stopping tablet replica
I20260202 15:07:08.049284 11451 raft_consensus.cc:2243] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.049868 11451 raft_consensus.cc:2272] T 1a70ec9fde0b4e8a896aed629584f8cb P 4d1008aec55e4adf85c21ea22e0ee0c5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.132310 11451 tablet_server.cc:195] TabletServer@127.11.46.193:0 shutdown complete.
W20260202 15:07:08.152697 11701 consensus_peers.cc:597] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 -> Peer 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249): Couldn't send request to peer 4d1008aec55e4adf85c21ea22e0ee0c5. Status: Network error: Client connection negotiation failed: client connection to 127.11.46.193:43249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:08.153767 11451 tablet_server.cc:178] TabletServer@127.11.46.194:0 shutting down...
W20260202 15:07:08.172528 11626 consensus_peers.cc:597] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 -> Peer 4d1008aec55e4adf85c21ea22e0ee0c5 (127.11.46.193:43249): Couldn't send request to peer 4d1008aec55e4adf85c21ea22e0ee0c5. Status: Network error: Client connection negotiation failed: client connection to 127.11.46.193:43249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:08.184793 11451 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260202 15:07:08.185458 11451 tablet_replica.cc:333] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71: stopping tablet replica
I20260202 15:07:08.186055 11451 raft_consensus.cc:2243] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.186476 11451 raft_consensus.cc:2272] T 892cc90953b1478c864856ee708597c5 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.188215 11451 tablet_replica.cc:333] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71: stopping tablet replica
I20260202 15:07:08.188665 11451 raft_consensus.cc:2243] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.189049 11451 raft_consensus.cc:2272] T 28f196b3798944459a69108f9d93b0d7 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.190862 11451 tablet_replica.cc:333] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71: stopping tablet replica
I20260202 15:07:08.192443 11451 raft_consensus.cc:2243] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.196549 11451 raft_consensus.cc:2272] T dd69d5f12dd2412089343e659c775418 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.198405 11451 tablet_replica.cc:333] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71: stopping tablet replica
I20260202 15:07:08.198966 11451 raft_consensus.cc:2243] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.199424 11451 raft_consensus.cc:2272] T fdf8b5e061f54d348072cddc0fbc2cda P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.201077 11451 tablet_replica.cc:333] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71: stopping tablet replica
I20260202 15:07:08.201503 11451 raft_consensus.cc:2243] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.202304 11451 raft_consensus.cc:2272] T 21c950e2b059499a87376daacd8a7c14 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.203841 11451 tablet_replica.cc:333] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71: stopping tablet replica
I20260202 15:07:08.204391 11451 raft_consensus.cc:2243] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.204784 11451 raft_consensus.cc:2272] T 9b4a87d39cb14268991cb083677738d2 P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.206465 11451 tablet_replica.cc:333] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71: stopping tablet replica
I20260202 15:07:08.206990 11451 raft_consensus.cc:2243] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.207402 11451 raft_consensus.cc:2272] T 90f30fdf5d624adda37324efe532a47d P 9079295b8e5e40dfbc4d95468b0a0f71 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.209012 11451 tablet_replica.cc:333] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71: stopping tablet replica
I20260202 15:07:08.209443 11451 raft_consensus.cc:2243] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.210347 11451 raft_consensus.cc:2272] T 5e50d403143c44cbb571fb0fae1a0d4e P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.211818 11451 tablet_replica.cc:333] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71: stopping tablet replica
I20260202 15:07:08.212195 11451 raft_consensus.cc:2243] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.212550 11451 raft_consensus.cc:2272] T 1a70ec9fde0b4e8a896aed629584f8cb P 9079295b8e5e40dfbc4d95468b0a0f71 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.241565 11451 tablet_server.cc:195] TabletServer@127.11.46.194:0 shutdown complete.
I20260202 15:07:08.260634 11451 tablet_server.cc:178] TabletServer@127.11.46.195:0 shutting down...
I20260202 15:07:08.298734 11451 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260202 15:07:08.299440 11451 tablet_replica.cc:333] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9: stopping tablet replica
I20260202 15:07:08.300113 11451 raft_consensus.cc:2243] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.300753 11451 raft_consensus.cc:2272] T dd69d5f12dd2412089343e659c775418 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.302489 11451 tablet_replica.cc:333] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9: stopping tablet replica
I20260202 15:07:08.303109 11451 raft_consensus.cc:2243] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 2 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.303893 11451 raft_consensus.cc:2272] T 892cc90953b1478c864856ee708597c5 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.305430 11451 tablet_replica.cc:333] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9: stopping tablet replica
I20260202 15:07:08.305902 11451 raft_consensus.cc:2243] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.306376 11451 raft_consensus.cc:2272] T fdf8b5e061f54d348072cddc0fbc2cda P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.307979 11451 tablet_replica.cc:333] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9: stopping tablet replica
I20260202 15:07:08.308480 11451 raft_consensus.cc:2243] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 3 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.309247 11451 raft_consensus.cc:2272] T 28f196b3798944459a69108f9d93b0d7 P 1fcf043552924663b3127821864eb1e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.310781 11451 tablet_replica.cc:333] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9: stopping tablet replica
I20260202 15:07:08.311213 11451 raft_consensus.cc:2243] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.311599 11451 raft_consensus.cc:2272] T 21c950e2b059499a87376daacd8a7c14 P 1fcf043552924663b3127821864eb1e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.313167 11451 tablet_replica.cc:333] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9: stopping tablet replica
I20260202 15:07:08.313665 11451 raft_consensus.cc:2243] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.314208 11451 raft_consensus.cc:2272] T 9b4a87d39cb14268991cb083677738d2 P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.315599 11451 tablet_replica.cc:333] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9: stopping tablet replica
I20260202 15:07:08.315995 11451 raft_consensus.cc:2243] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.316416 11451 raft_consensus.cc:2272] T 90f30fdf5d624adda37324efe532a47d P 1fcf043552924663b3127821864eb1e9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.317922 11451 tablet_replica.cc:333] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9: stopping tablet replica
I20260202 15:07:08.318396 11451 raft_consensus.cc:2243] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:08.318877 11451 raft_consensus.cc:2272] T 5e50d403143c44cbb571fb0fae1a0d4e P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.320372 11451 tablet_replica.cc:333] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9: stopping tablet replica
I20260202 15:07:08.320786 11451 raft_consensus.cc:2243] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 4 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.321480 11451 raft_consensus.cc:2272] T 1a70ec9fde0b4e8a896aed629584f8cb P 1fcf043552924663b3127821864eb1e9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.362155 11451 tablet_server.cc:195] TabletServer@127.11.46.195:0 shutdown complete.
I20260202 15:07:08.379180 11451 master.cc:561] Master@127.11.46.254:37841 shutting down...
I20260202 15:07:08.568113 11451 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:07:08.568686 11451 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:08.568986 11451 tablet_replica.cc:333] T 00000000000000000000000000000000 P c0393909e56d448daae90116c1c00daf: stopping tablet replica
I20260202 15:07:08.601687 11451 master.cc:583] Master@127.11.46.254:37841 shutdown complete.
1fcf043552924663b3127821864eb1e9 1
4d1008aec55e4adf85c21ea22e0ee0c5 4
9079295b8e5e40dfbc4d95468b0a0f71 4
1fcf043552924663b3127821864eb1e9 3
4d1008aec55e4adf85c21ea22e0ee0c5 3
9079295b8e5e40dfbc4d95468b0a0f71 3
1fcf043552924663b3127821864eb1e9 1
4d1008aec55e4adf85c21ea22e0ee0c5 0
9079295b8e5e40dfbc4d95468b0a0f71 8
1fcf043552924663b3127821864eb1e9 3
4d1008aec55e4adf85c21ea22e0ee0c5 3
9079295b8e5e40dfbc4d95468b0a0f71 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30850 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260202 15:07:08.638381 11451 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.46.254:37587
I20260202 15:07:08.639381 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:07:08.644249 11982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:08.644326 11983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:08.645588 11451 server_base.cc:1060] running on GCE node
W20260202 15:07:08.645788 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:08.646864 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:07:08.647048 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:07:08.647158 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044828647148 us; error 0 us; skew 500 ppm
I20260202 15:07:08.647617 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:07:08.649781 11451 webserver.cc:492] Webserver started at http://127.11.46.254:36783/ using document root <none> and password file <none>
I20260202 15:07:08.650208 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:07:08.650372 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:07:08.650566 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:07:08.651777 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/master-0-root/instance:
uuid: "52f9dbde7f6f418e96c88e824b0f3ba5"
format_stamp: "Formatted at 2026-02-02 15:07:08 on dist-test-slave-rflp"
I20260202 15:07:08.655689 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260202 15:07:08.658646 11990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:08.659348 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260202 15:07:08.659608 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/master-0-root
uuid: "52f9dbde7f6f418e96c88e824b0f3ba5"
format_stamp: "Formatted at 2026-02-02 15:07:08 on dist-test-slave-rflp"
I20260202 15:07:08.659818 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:07:08.674336 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:07:08.675441 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:07:08.710847 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.254:37587
I20260202 15:07:08.710958 12041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.254:37587 every 8 connection(s)
I20260202 15:07:08.714299 12042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:08.723167 12042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5: Bootstrap starting.
I20260202 15:07:08.727330 12042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:08.731244 12042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5: No bootstrap required, opened a new log
I20260202 15:07:08.733139 12042 raft_consensus.cc:359] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f9dbde7f6f418e96c88e824b0f3ba5" member_type: VOTER }
I20260202 15:07:08.733525 12042 raft_consensus.cc:385] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:08.733778 12042 raft_consensus.cc:740] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f9dbde7f6f418e96c88e824b0f3ba5, State: Initialized, Role: FOLLOWER
I20260202 15:07:08.734412 12042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f9dbde7f6f418e96c88e824b0f3ba5" member_type: VOTER }
I20260202 15:07:08.734885 12042 raft_consensus.cc:399] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:07:08.735127 12042 raft_consensus.cc:493] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:07:08.735440 12042 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:08.738807 12042 raft_consensus.cc:515] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f9dbde7f6f418e96c88e824b0f3ba5" member_type: VOTER }
I20260202 15:07:08.739252 12042 leader_election.cc:304] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52f9dbde7f6f418e96c88e824b0f3ba5; no voters:
I20260202 15:07:08.740191 12042 leader_election.cc:290] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:07:08.740592 12045 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:08.741847 12045 raft_consensus.cc:697] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 1 LEADER]: Becoming Leader. State: Replica: 52f9dbde7f6f418e96c88e824b0f3ba5, State: Running, Role: LEADER
I20260202 15:07:08.742530 12045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f9dbde7f6f418e96c88e824b0f3ba5" member_type: VOTER }
I20260202 15:07:08.743221 12042 sys_catalog.cc:565] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:07:08.747843 12047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52f9dbde7f6f418e96c88e824b0f3ba5. Latest consensus state: current_term: 1 leader_uuid: "52f9dbde7f6f418e96c88e824b0f3ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f9dbde7f6f418e96c88e824b0f3ba5" member_type: VOTER } }
I20260202 15:07:08.748451 12047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [sys.catalog]: This master's current role is: LEADER
I20260202 15:07:08.748709 12046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52f9dbde7f6f418e96c88e824b0f3ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f9dbde7f6f418e96c88e824b0f3ba5" member_type: VOTER } }
I20260202 15:07:08.749295 12046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [sys.catalog]: This master's current role is: LEADER
I20260202 15:07:08.750502 12053 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:07:08.755225 12053 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:07:08.757617 11451 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260202 15:07:08.763036 12053 catalog_manager.cc:1357] Generated new cluster ID: a4858ea01c954175af566b44c18aaff2
I20260202 15:07:08.763288 12053 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:07:08.777287 12053 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:07:08.778319 12053 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:07:08.789965 12053 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5: Generated new TSK 0
I20260202 15:07:08.790514 12053 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:07:08.823876 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:07:08.829759 12063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:08.830876 12064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:08.832363 12066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:08.833276 11451 server_base.cc:1060] running on GCE node
I20260202 15:07:08.834117 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:07:08.834301 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:07:08.834448 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044828834412 us; error 0 us; skew 500 ppm
I20260202 15:07:08.834935 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:07:08.836920 11451 webserver.cc:492] Webserver started at http://127.11.46.193:40545/ using document root <none> and password file <none>
I20260202 15:07:08.837394 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:07:08.837592 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:07:08.837867 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:07:08.838796 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-0-root/instance:
uuid: "93e91e2eb8834e7b8d844526093e59d6"
format_stamp: "Formatted at 2026-02-02 15:07:08 on dist-test-slave-rflp"
I20260202 15:07:08.842367 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260202 15:07:08.845199 12071 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:08.845824 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:07:08.846099 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-0-root
uuid: "93e91e2eb8834e7b8d844526093e59d6"
format_stamp: "Formatted at 2026-02-02 15:07:08 on dist-test-slave-rflp"
I20260202 15:07:08.846391 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:07:08.862917 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:07:08.864043 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:07:08.865396 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:07:08.867439 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:07:08.867643 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:08.867902 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:07:08.868093 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:08.908787 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.193:40833
I20260202 15:07:08.908885 12133 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.193:40833 every 8 connection(s)
I20260202 15:07:08.912832 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:07:08.919256 12138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:08.920039 12139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:08.922583 11451 server_base.cc:1060] running on GCE node
W20260202 15:07:08.923938 12141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:08.924729 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:07:08.924934 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:07:08.925113 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044828925101 us; error 0 us; skew 500 ppm
I20260202 15:07:08.925581 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:07:08.927659 11451 webserver.cc:492] Webserver started at http://127.11.46.194:32871/ using document root <none> and password file <none>
I20260202 15:07:08.928193 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:07:08.928421 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:07:08.928675 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:07:08.929605 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-1-root/instance:
uuid: "3270daa0942c4603b01ef4ddab75ec12"
format_stamp: "Formatted at 2026-02-02 15:07:08 on dist-test-slave-rflp"
I20260202 15:07:08.931005 12134 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37587
I20260202 15:07:08.931334 12134 heartbeater.cc:461] Registering TS with master...
I20260202 15:07:08.931950 12134 heartbeater.cc:507] Master 127.11.46.254:37587 requested a full tablet report, sending...
I20260202 15:07:08.933832 12007 ts_manager.cc:194] Registered new tserver with Master: 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833)
I20260202 15:07:08.934478 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260202 15:07:08.935410 12007 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35506
I20260202 15:07:08.938002 12146 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:08.938843 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:07:08.939133 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-1-root
uuid: "3270daa0942c4603b01ef4ddab75ec12"
format_stamp: "Formatted at 2026-02-02 15:07:08 on dist-test-slave-rflp"
I20260202 15:07:08.939424 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:07:08.956470 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:07:08.957777 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:07:08.959038 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:07:08.960914 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:07:08.961057 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:08.961300 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:07:08.961483 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:08.997030 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.194:43227
I20260202 15:07:08.997130 12208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.194:43227 every 8 connection(s)
I20260202 15:07:09.001178 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:07:09.008831 12212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:09.011441 12213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:09.014851 12215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:09.014979 12209 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37587
I20260202 15:07:09.015218 11451 server_base.cc:1060] running on GCE node
I20260202 15:07:09.015405 12209 heartbeater.cc:461] Registering TS with master...
I20260202 15:07:09.016431 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260202 15:07:09.016427 12209 heartbeater.cc:507] Master 127.11.46.254:37587 requested a full tablet report, sending...
W20260202 15:07:09.016959 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:07:09.017175 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044829017155 us; error 0 us; skew 500 ppm
I20260202 15:07:09.017902 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:07:09.019176 12007 ts_manager.cc:194] Registered new tserver with Master: 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:09.020666 11451 webserver.cc:492] Webserver started at http://127.11.46.195:42409/ using document root <none> and password file <none>
I20260202 15:07:09.020951 12007 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35512
I20260202 15:07:09.021238 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:07:09.021456 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:07:09.021750 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:07:09.023212 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-2-root/instance:
uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
format_stamp: "Formatted at 2026-02-02 15:07:09 on dist-test-slave-rflp"
I20260202 15:07:09.029202 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260202 15:07:09.033265 12220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:09.034106 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:07:09.034389 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-2-root
uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
format_stamp: "Formatted at 2026-02-02 15:07:09 on dist-test-slave-rflp"
I20260202 15:07:09.034674 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1770044797646468-11451-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:07:09.053208 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:07:09.054193 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:07:09.055442 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:07:09.057337 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:07:09.057530 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:09.057780 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:07:09.057912 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:09.098559 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.195:40889
I20260202 15:07:09.098676 12282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.195:40889 every 8 connection(s)
I20260202 15:07:09.111572 12283 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37587
I20260202 15:07:09.111963 12283 heartbeater.cc:461] Registering TS with master...
I20260202 15:07:09.112668 12283 heartbeater.cc:507] Master 127.11.46.254:37587 requested a full tablet report, sending...
I20260202 15:07:09.114440 12007 ts_manager.cc:194] Registered new tserver with Master: 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:09.115228 11451 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013550602s
I20260202 15:07:09.115993 12007 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35514
I20260202 15:07:09.140648 12007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35518:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:07:09.143659 12007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:07:09.232817 12099 tablet_service.cc:1505] Processing CreateTablet for tablet d38bd74acf9241bc8aec9c60f7ab3924 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260202 15:07:09.234294 12099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38bd74acf9241bc8aec9c60f7ab3924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.234181 12248 tablet_service.cc:1505] Processing CreateTablet for tablet d38bd74acf9241bc8aec9c60f7ab3924 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260202 15:07:09.235174 12248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38bd74acf9241bc8aec9c60f7ab3924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.235879 12098 tablet_service.cc:1505] Processing CreateTablet for tablet 68bb2b68bcfb46aca45082c97b6b4ace (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260202 15:07:09.237216 12098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68bb2b68bcfb46aca45082c97b6b4ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.239599 12247 tablet_service.cc:1505] Processing CreateTablet for tablet 68bb2b68bcfb46aca45082c97b6b4ace (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260202 15:07:09.239866 12097 tablet_service.cc:1505] Processing CreateTablet for tablet c0ebeba16c704c3eb18cacfbfc74d95e (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260202 15:07:09.240602 12247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68bb2b68bcfb46aca45082c97b6b4ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.241219 12097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ebeba16c704c3eb18cacfbfc74d95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.243886 12096 tablet_service.cc:1505] Processing CreateTablet for tablet 0fa1440387614a6a9c30b73df4c211fd (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260202 15:07:09.244094 12094 tablet_service.cc:1505] Processing CreateTablet for tablet c74740e8b5ef4ff8b61d37b275dd4dc1 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260202 15:07:09.245288 12096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa1440387614a6a9c30b73df4c211fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.245844 12245 tablet_service.cc:1505] Processing CreateTablet for tablet 0fa1440387614a6a9c30b73df4c211fd (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260202 15:07:09.247236 12245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa1440387614a6a9c30b73df4c211fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.245919 12095 tablet_service.cc:1505] Processing CreateTablet for tablet 11f27edb4a944789ade0124dbf15d314 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260202 15:07:09.249198 12094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c74740e8b5ef4ff8b61d37b275dd4dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.243938 12246 tablet_service.cc:1505] Processing CreateTablet for tablet c0ebeba16c704c3eb18cacfbfc74d95e (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260202 15:07:09.250224 12244 tablet_service.cc:1505] Processing CreateTablet for tablet 11f27edb4a944789ade0124dbf15d314 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260202 15:07:09.253446 12244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f27edb4a944789ade0124dbf15d314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.256469 12246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ebeba16c704c3eb18cacfbfc74d95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.259454 12241 tablet_service.cc:1505] Processing CreateTablet for tablet 58af84d12bfe4f359253736f8bbdce4b (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260202 15:07:09.260509 12241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58af84d12bfe4f359253736f8bbdce4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.262961 12093 tablet_service.cc:1505] Processing CreateTablet for tablet 763ffb7935a149728615aec0b0f953f7 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260202 15:07:09.263883 12239 tablet_service.cc:1505] Processing CreateTablet for tablet 05b935b29d684295a2d3587ff36f921e (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260202 15:07:09.265081 12239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b935b29d684295a2d3587ff36f921e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.265317 12092 tablet_service.cc:1505] Processing CreateTablet for tablet 58af84d12bfe4f359253736f8bbdce4b (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260202 15:07:09.267407 12091 tablet_service.cc:1505] Processing CreateTablet for tablet 9b2521523b414554957817815d71e97f (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260202 15:07:09.267809 12242 tablet_service.cc:1505] Processing CreateTablet for tablet 763ffb7935a149728615aec0b0f953f7 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260202 15:07:09.269119 12242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763ffb7935a149728615aec0b0f953f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.269420 12090 tablet_service.cc:1505] Processing CreateTablet for tablet 05b935b29d684295a2d3587ff36f921e (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260202 15:07:09.271845 12240 tablet_service.cc:1505] Processing CreateTablet for tablet 9b2521523b414554957817815d71e97f (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260202 15:07:09.273300 12240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2521523b414554957817815d71e97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.256470 12243 tablet_service.cc:1505] Processing CreateTablet for tablet c74740e8b5ef4ff8b61d37b275dd4dc1 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260202 15:07:09.276501 12243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c74740e8b5ef4ff8b61d37b275dd4dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.278592 12095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f27edb4a944789ade0124dbf15d314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.283032 12090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b935b29d684295a2d3587ff36f921e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.294365 12092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58af84d12bfe4f359253736f8bbdce4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.295480 12093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763ffb7935a149728615aec0b0f953f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.303097 12091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2521523b414554957817815d71e97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.345055 12173 tablet_service.cc:1505] Processing CreateTablet for tablet 68bb2b68bcfb46aca45082c97b6b4ace (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260202 15:07:09.346386 12173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68bb2b68bcfb46aca45082c97b6b4ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.347077 12167 tablet_service.cc:1505] Processing CreateTablet for tablet 58af84d12bfe4f359253736f8bbdce4b (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260202 15:07:09.351861 12171 tablet_service.cc:1505] Processing CreateTablet for tablet 0fa1440387614a6a9c30b73df4c211fd (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260202 15:07:09.347923 12172 tablet_service.cc:1505] Processing CreateTablet for tablet c0ebeba16c704c3eb18cacfbfc74d95e (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260202 15:07:09.347929 12174 tablet_service.cc:1505] Processing CreateTablet for tablet d38bd74acf9241bc8aec9c60f7ab3924 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260202 15:07:09.354228 12165 tablet_service.cc:1505] Processing CreateTablet for tablet 05b935b29d684295a2d3587ff36f921e (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260202 15:07:09.351821 12166 tablet_service.cc:1505] Processing CreateTablet for tablet 9b2521523b414554957817815d71e97f (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260202 15:07:09.357084 12168 tablet_service.cc:1505] Processing CreateTablet for tablet 763ffb7935a149728615aec0b0f953f7 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260202 15:07:09.359983 12170 tablet_service.cc:1505] Processing CreateTablet for tablet 11f27edb4a944789ade0124dbf15d314 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260202 15:07:09.362726 12169 tablet_service.cc:1505] Processing CreateTablet for tablet c74740e8b5ef4ff8b61d37b275dd4dc1 (DEFAULT_TABLE table=test-workload [id=54c527c95c0f46f1aa9a6b5ea0b40aec]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260202 15:07:09.366847 12167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58af84d12bfe4f359253736f8bbdce4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.375308 12166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2521523b414554957817815d71e97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.379055 12174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38bd74acf9241bc8aec9c60f7ab3924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.383069 12170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f27edb4a944789ade0124dbf15d314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.385840 12169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c74740e8b5ef4ff8b61d37b275dd4dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.387029 12171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa1440387614a6a9c30b73df4c211fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.391022 12165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b935b29d684295a2d3587ff36f921e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.394994 12168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763ffb7935a149728615aec0b0f953f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.403054 12172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ebeba16c704c3eb18cacfbfc74d95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:09.443145 12299 tablet_bootstrap.cc:492] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:09.448952 12299 tablet_bootstrap.cc:654] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.469265 12298 tablet_bootstrap.cc:492] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.480685 12298 tablet_bootstrap.cc:654] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.487433 12300 tablet_bootstrap.cc:492] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:09.494442 12299 tablet_bootstrap.cc:492] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:09.494972 12299 ts_tablet_manager.cc:1403] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.055s user 0.014s sys 0.013s
I20260202 15:07:09.497829 12299 raft_consensus.cc:359] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.498724 12299 raft_consensus.cc:385] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.499076 12299 raft_consensus.cc:740] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.504345 12299 consensus_queue.cc:260] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.506982 12300 tablet_bootstrap.cc:654] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.509938 12283 heartbeater.cc:499] Master 127.11.46.254:37587 was elected leader, sending a full tablet report...
I20260202 15:07:09.510371 12298 tablet_bootstrap.cc:492] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.510563 12299 ts_tablet_manager.cc:1434] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.015s user 0.010s sys 0.000s
I20260202 15:07:09.510826 12298 ts_tablet_manager.cc:1403] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.018s
I20260202 15:07:09.511660 12299 tablet_bootstrap.cc:492] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:09.515089 12298 raft_consensus.cc:359] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } }
I20260202 15:07:09.516140 12298 raft_consensus.cc:385] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.516637 12298 raft_consensus.cc:740] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.518579 12299 tablet_bootstrap.cc:654] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.518474 12298 consensus_queue.cc:260] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } }
I20260202 15:07:09.527631 12134 heartbeater.cc:499] Master 127.11.46.254:37587 was elected leader, sending a full tablet report...
I20260202 15:07:09.535241 12298 ts_tablet_manager.cc:1434] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.024s user 0.011s sys 0.004s
I20260202 15:07:09.537159 12298 tablet_bootstrap.cc:492] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.533564 12299 tablet_bootstrap.cc:492] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:09.545836 12299 ts_tablet_manager.cc:1403] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.004s
I20260202 15:07:09.550101 12300 tablet_bootstrap.cc:492] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:09.551220 12298 tablet_bootstrap.cc:654] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.551565 12300 ts_tablet_manager.cc:1403] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.065s user 0.018s sys 0.028s
I20260202 15:07:09.554039 12299 raft_consensus.cc:359] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.554845 12299 raft_consensus.cc:385] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.555176 12299 raft_consensus.cc:740] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.555265 12300 raft_consensus.cc:359] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.556042 12300 raft_consensus.cc:385] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.556465 12300 raft_consensus.cc:740] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.556077 12299 consensus_queue.cc:260] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.557919 12300 consensus_queue.cc:260] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.560257 12299 ts_tablet_manager.cc:1434] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260202 15:07:09.562197 12299 tablet_bootstrap.cc:492] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:09.568881 12209 heartbeater.cc:499] Master 127.11.46.254:37587 was elected leader, sending a full tablet report...
I20260202 15:07:09.570214 12300 ts_tablet_manager.cc:1434] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.017s user 0.012s sys 0.002s
I20260202 15:07:09.571310 12300 tablet_bootstrap.cc:492] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:09.579298 12300 tablet_bootstrap.cc:654] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.588546 12299 tablet_bootstrap.cc:654] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.600608 12298 tablet_bootstrap.cc:492] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.601202 12298 ts_tablet_manager.cc:1403] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.065s user 0.021s sys 0.022s
I20260202 15:07:09.604274 12300 tablet_bootstrap.cc:492] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:09.604274 12298 raft_consensus.cc:359] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.605074 12300 ts_tablet_manager.cc:1403] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.008s
I20260202 15:07:09.605304 12298 raft_consensus.cc:385] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.605504 12299 tablet_bootstrap.cc:492] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:09.605736 12298 raft_consensus.cc:740] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.606182 12299 ts_tablet_manager.cc:1403] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.010s
I20260202 15:07:09.606947 12298 consensus_queue.cc:260] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.608713 12300 raft_consensus.cc:359] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.609483 12300 raft_consensus.cc:385] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.609458 12299 raft_consensus.cc:359] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.610030 12300 raft_consensus.cc:740] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.610343 12299 raft_consensus.cc:385] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.610749 12299 raft_consensus.cc:740] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.611315 12300 consensus_queue.cc:260] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.611581 12299 consensus_queue.cc:260] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.613368 12298 ts_tablet_manager.cc:1434] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260202 15:07:09.613451 12300 ts_tablet_manager.cc:1434] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260202 15:07:09.613852 12299 ts_tablet_manager.cc:1434] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260202 15:07:09.614347 12300 tablet_bootstrap.cc:492] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:09.614351 12298 tablet_bootstrap.cc:492] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.614806 12299 tablet_bootstrap.cc:492] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:09.623703 12299 tablet_bootstrap.cc:654] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.624343 12300 tablet_bootstrap.cc:654] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.628604 12298 tablet_bootstrap.cc:654] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.636297 12299 tablet_bootstrap.cc:492] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:09.637625 12299 ts_tablet_manager.cc:1403] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.003s
I20260202 15:07:09.638808 12300 tablet_bootstrap.cc:492] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:09.640021 12300 ts_tablet_manager.cc:1403] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.003s
I20260202 15:07:09.642045 12299 raft_consensus.cc:359] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.643088 12299 raft_consensus.cc:385] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.643528 12299 raft_consensus.cc:740] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.643880 12300 raft_consensus.cc:359] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.644791 12300 raft_consensus.cc:385] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.645119 12300 raft_consensus.cc:740] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.644553 12299 consensus_queue.cc:260] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.645867 12300 consensus_queue.cc:260] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.647400 12299 ts_tablet_manager.cc:1434] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260202 15:07:09.648391 12300 ts_tablet_manager.cc:1434] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260202 15:07:09.648619 12299 tablet_bootstrap.cc:492] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:09.649278 12300 tablet_bootstrap.cc:492] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:09.650095 12303 raft_consensus.cc:493] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:09.650573 12303 raft_consensus.cc:515] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.655690 12299 tablet_bootstrap.cc:654] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.656584 12300 tablet_bootstrap.cc:654] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.669391 12298 tablet_bootstrap.cc:492] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.675529 12298 ts_tablet_manager.cc:1403] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.061s user 0.016s sys 0.007s
I20260202 15:07:09.679754 12298 raft_consensus.cc:359] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.680464 12298 raft_consensus.cc:385] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.680753 12298 raft_consensus.cc:740] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.681428 12298 consensus_queue.cc:260] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.666997 12299 tablet_bootstrap.cc:492] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:09.682967 12299 ts_tablet_manager.cc:1403] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.005s
I20260202 15:07:09.666777 12303 leader_election.cc:290] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:09.686585 12299 raft_consensus.cc:359] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.690481 12298 ts_tablet_manager.cc:1434] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.015s user 0.006s sys 0.002s
I20260202 15:07:09.687598 12299 raft_consensus.cc:385] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.692984 12298 tablet_bootstrap.cc:492] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.695063 12299 raft_consensus.cc:740] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.696615 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6" is_pre_election: true
I20260202 15:07:09.698378 12109 raft_consensus.cc:2468] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 0.
I20260202 15:07:09.699963 12300 tablet_bootstrap.cc:492] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:09.700800 12300 ts_tablet_manager.cc:1403] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.052s user 0.037s sys 0.011s
I20260202 15:07:09.700419 12299 consensus_queue.cc:260] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.705103 12300 raft_consensus.cc:359] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.706176 12300 raft_consensus.cc:385] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.706811 12300 raft_consensus.cc:740] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.708107 12223 leader_election.cc:304] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:09.708026 12300 consensus_queue.cc:260] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.709775 12308 raft_consensus.cc:493] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:09.710479 12308 raft_consensus.cc:515] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.712178 12303 raft_consensus.cc:2804] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:09.712692 12303 raft_consensus.cc:493] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:09.713166 12303 raft_consensus.cc:3060] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:09.716724 12300 ts_tablet_manager.cc:1434] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260202 15:07:09.717677 12300 tablet_bootstrap.cc:492] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:09.719799 12308 leader_election.cc:290] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:09.722482 12303 raft_consensus.cc:515] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.724721 12300 tablet_bootstrap.cc:654] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.725562 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:09.726840 12109 raft_consensus.cc:3060] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:09.729499 12299 ts_tablet_manager.cc:1434] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.046s user 0.005s sys 0.005s
I20260202 15:07:09.731340 12303 leader_election.cc:290] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:09.733250 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12" is_pre_election: true
I20260202 15:07:09.733696 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:09.734124 12184 raft_consensus.cc:2468] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 0.
I20260202 15:07:09.752605 12109 raft_consensus.cc:2468] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 1.
W20260202 15:07:09.753907 12055 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:07:09.754374 12223 leader_election.cc:304] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 93e91e2eb8834e7b8d844526093e59d6; no voters: 3270daa0942c4603b01ef4ddab75ec12
I20260202 15:07:09.755183 12299 tablet_bootstrap.cc:492] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:09.755738 12303 raft_consensus.cc:2804] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:09.759400 12298 tablet_bootstrap.cc:654] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.761685 12308 raft_consensus.cc:493] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:09.762338 12308 raft_consensus.cc:515] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.764606 12308 leader_election.cc:290] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:09.765133 12300 tablet_bootstrap.cc:492] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:09.765647 12300 ts_tablet_manager.cc:1403] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.021s
I20260202 15:07:09.766566 12299 tablet_bootstrap.cc:654] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.766851 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b2521523b414554957817815d71e97f" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6" is_pre_election: true
I20260202 15:07:09.768734 12108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6" is_pre_election: true
I20260202 15:07:09.769165 12300 raft_consensus.cc:359] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.769914 12300 raft_consensus.cc:385] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260202 15:07:09.770221 12149 leader_election.cc:343] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:09.770931 12149 leader_election.cc:343] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:09.775136 12303 raft_consensus.cc:697] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Becoming Leader. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:09.776166 12303 consensus_queue.cc:237] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.777556 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" is_pre_election: true
I20260202 15:07:09.778436 12257 raft_consensus.cc:2468] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 0.
I20260202 15:07:09.770233 12300 raft_consensus.cc:740] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.779887 12150 leader_election.cc:304] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters: 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:09.780264 12300 consensus_queue.cc:260] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.780789 12308 raft_consensus.cc:2804] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:09.781349 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b2521523b414554957817815d71e97f" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" is_pre_election: true
I20260202 15:07:09.781756 12308 raft_consensus.cc:493] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:09.782084 12308 raft_consensus.cc:3060] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:09.781975 12258 raft_consensus.cc:2468] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 0.
I20260202 15:07:09.782738 12300 ts_tablet_manager.cc:1434] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.016s user 0.002s sys 0.005s
I20260202 15:07:09.783747 12300 tablet_bootstrap.cc:492] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:09.785830 12150 leader_election.cc:304] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters: 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:09.787034 12318 raft_consensus.cc:2804] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:09.787387 12318 raft_consensus.cc:493] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:09.787699 12318 raft_consensus.cc:3060] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:09.791559 12308 raft_consensus.cc:515] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.794152 12300 tablet_bootstrap.cc:654] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.795034 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
W20260202 15:07:09.796975 12149 leader_election.cc:343] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:09.799372 12308 leader_election.cc:290] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:09.800784 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:09.801522 12258 raft_consensus.cc:3060] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:09.802484 12298 tablet_bootstrap.cc:492] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.803133 12298 ts_tablet_manager.cc:1403] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.110s user 0.024s sys 0.000s
I20260202 15:07:09.807164 12298 raft_consensus.cc:359] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.808257 12298 raft_consensus.cc:385] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.808709 12298 raft_consensus.cc:740] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.809751 12298 consensus_queue.cc:260] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.812211 12318 raft_consensus.cc:515] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.813520 12298 ts_tablet_manager.cc:1434] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260202 15:07:09.814834 12298 tablet_bootstrap.cc:492] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.814918 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b2521523b414554957817815d71e97f" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
W20260202 15:07:09.816700 12149 leader_election.cc:343] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:09.817631 12318 leader_election.cc:290] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:09.818420 12258 raft_consensus.cc:2468] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 1.
I20260202 15:07:09.818787 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b2521523b414554957817815d71e97f" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:09.820052 12257 raft_consensus.cc:3060] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:09.821262 12150 leader_election.cc:304] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters: 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:09.824910 12006 catalog_manager.cc:5654] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:09.828153 12298 tablet_bootstrap.cc:654] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.837282 12299 tablet_bootstrap.cc:492] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:09.837981 12299 ts_tablet_manager.cc:1403] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.083s user 0.027s sys 0.008s
I20260202 15:07:09.841048 12298 tablet_bootstrap.cc:492] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.841897 12298 ts_tablet_manager.cc:1403] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.004s
I20260202 15:07:09.844710 12318 raft_consensus.cc:2804] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:09.845382 12298 raft_consensus.cc:359] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.846318 12318 raft_consensus.cc:697] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:09.846495 12298 raft_consensus.cc:385] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.847129 12298 raft_consensus.cc:740] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.847544 12318 consensus_queue.cc:237] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.848256 12298 consensus_queue.cc:260] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.849809 12257 raft_consensus.cc:2468] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 1.
I20260202 15:07:09.850596 12299 raft_consensus.cc:359] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.850948 12150 leader_election.cc:304] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters: 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:09.851228 12299 raft_consensus.cc:385] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.851511 12299 raft_consensus.cc:740] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.851943 12308 raft_consensus.cc:2804] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:09.852079 12299 consensus_queue.cc:260] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.854342 12299 ts_tablet_manager.cc:1434] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260202 15:07:09.855439 12299 tablet_bootstrap.cc:492] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:09.857792 12298 ts_tablet_manager.cc:1434] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260202 15:07:09.858798 12298 tablet_bootstrap.cc:492] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.860584 12303 raft_consensus.cc:493] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:09.861014 12303 raft_consensus.cc:515] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.851218 12300 tablet_bootstrap.cc:492] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:09.863498 12300 ts_tablet_manager.cc:1403] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.030s
I20260202 15:07:09.864148 12308 raft_consensus.cc:697] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:09.864303 12303 leader_election.cc:290] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:09.864384 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12" is_pre_election: true
I20260202 15:07:09.865131 12308 consensus_queue.cc:237] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.865928 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6" is_pre_election: true
I20260202 15:07:09.862498 12299 tablet_bootstrap.cc:654] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:09.867473 12223 leader_election.cc:343] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:09.868585 12221 leader_election.cc:343] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:09.868928 12298 tablet_bootstrap.cc:654] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.868980 12221 leader_election.cc:304] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61; no voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:09.869202 12300 raft_consensus.cc:359] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.869828 12300 raft_consensus.cc:385] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.870069 12303 raft_consensus.cc:2749] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:09.870177 12300 raft_consensus.cc:740] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.871184 12300 consensus_queue.cc:260] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.875133 12300 ts_tablet_manager.cc:1434] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260202 15:07:09.879444 12300 tablet_bootstrap.cc:492] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:09.879944 12298 tablet_bootstrap.cc:492] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.880460 12298 ts_tablet_manager.cc:1403] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.008s
I20260202 15:07:09.882484 12298 raft_consensus.cc:359] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.883046 12298 raft_consensus.cc:385] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.883285 12298 raft_consensus.cc:740] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.883831 12298 consensus_queue.cc:260] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.884848 12300 tablet_bootstrap.cc:654] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.885658 12298 ts_tablet_manager.cc:1434] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260202 15:07:09.888075 12299 tablet_bootstrap.cc:492] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:09.888561 12299 ts_tablet_manager.cc:1403] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.002s
I20260202 15:07:09.890980 12299 raft_consensus.cc:359] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } }
I20260202 15:07:09.891587 12299 raft_consensus.cc:385] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.891881 12299 raft_consensus.cc:740] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.892423 12298 tablet_bootstrap.cc:492] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.893354 12305 raft_consensus.cc:493] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:09.892782 12004 catalog_manager.cc:5654] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "3270daa0942c4603b01ef4ddab75ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:09.893791 12305 raft_consensus.cc:515] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } }
I20260202 15:07:09.896059 12305 leader_election.cc:290] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:09.900275 12298 tablet_bootstrap.cc:654] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.906580 12298 tablet_bootstrap.cc:492] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.907203 12298 ts_tablet_manager.cc:1403] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260202 15:07:09.908820 12004 catalog_manager.cc:5654] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "3270daa0942c4603b01ef4ddab75ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:09.910171 12298 raft_consensus.cc:359] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.911093 12298 raft_consensus.cc:385] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.911507 12298 raft_consensus.cc:740] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.912436 12298 consensus_queue.cc:260] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.914896 12298 ts_tablet_manager.cc:1434] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260202 15:07:09.915928 12298 tablet_bootstrap.cc:492] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.892560 12299 consensus_queue.cc:260] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } }
I20260202 15:07:09.923311 12298 tablet_bootstrap.cc:654] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.925284 12299 ts_tablet_manager.cc:1434] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.036s user 0.001s sys 0.004s
I20260202 15:07:09.929277 12299 tablet_bootstrap.cc:492] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:09.940551 12300 tablet_bootstrap.cc:492] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:09.941125 12300 ts_tablet_manager.cc:1403] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.062s user 0.028s sys 0.032s
I20260202 15:07:09.943949 12300 raft_consensus.cc:359] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } }
I20260202 15:07:09.944590 12300 raft_consensus.cc:385] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.944684 12298 tablet_bootstrap.cc:492] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.944944 12300 raft_consensus.cc:740] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.945194 12298 ts_tablet_manager.cc:1403] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.013s
I20260202 15:07:09.945823 12300 consensus_queue.cc:260] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } }
I20260202 15:07:09.947552 12298 raft_consensus.cc:359] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.948283 12298 raft_consensus.cc:385] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.948563 12298 raft_consensus.cc:740] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.949218 12298 consensus_queue.cc:260] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.950971 12298 ts_tablet_manager.cc:1434] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:07:09.951990 12298 tablet_bootstrap.cc:492] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.953167 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38bd74acf9241bc8aec9c60f7ab3924" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12" is_pre_election: true
I20260202 15:07:09.953753 12183 raft_consensus.cc:2468] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 0.
I20260202 15:07:09.954952 12072 leader_election.cc:304] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:09.955430 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38bd74acf9241bc8aec9c60f7ab3924" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" is_pre_election: true
I20260202 15:07:09.955967 12257 raft_consensus.cc:2468] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 0.
I20260202 15:07:09.956648 12305 raft_consensus.cc:2804] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:09.956946 12298 tablet_bootstrap.cc:654] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.956987 12305 raft_consensus.cc:493] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:09.957237 12300 ts_tablet_manager.cc:1434] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.016s user 0.003s sys 0.002s
I20260202 15:07:09.957479 12305 raft_consensus.cc:3060] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:09.958199 12300 tablet_bootstrap.cc:492] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:09.964885 12298 tablet_bootstrap.cc:492] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.965344 12298 ts_tablet_manager.cc:1403] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260202 15:07:09.966298 12299 tablet_bootstrap.cc:654] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.962394 12305 raft_consensus.cc:515] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } }
I20260202 15:07:09.967700 12298 raft_consensus.cc:359] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.968346 12298 raft_consensus.cc:385] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.968638 12298 raft_consensus.cc:740] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.968725 12300 tablet_bootstrap.cc:654] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.968801 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38bd74acf9241bc8aec9c60f7ab3924" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:09.969348 12257 raft_consensus.cc:3060] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:09.969388 12298 consensus_queue.cc:260] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.972158 12298 ts_tablet_manager.cc:1434] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260202 15:07:09.972970 12298 tablet_bootstrap.cc:492] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6: Bootstrap starting.
I20260202 15:07:09.973510 12300 tablet_bootstrap.cc:492] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:09.973875 12300 ts_tablet_manager.cc:1403] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260202 15:07:09.974336 12257 raft_consensus.cc:2468] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 1.
I20260202 15:07:09.975457 12305 leader_election.cc:290] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 election: Requested vote from peers 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:09.976075 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38bd74acf9241bc8aec9c60f7ab3924" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:09.976610 12183 raft_consensus.cc:3060] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:09.977989 12298 tablet_bootstrap.cc:654] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:09.982914 12303 raft_consensus.cc:493] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:09.983510 12303 raft_consensus.cc:515] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.984139 12183 raft_consensus.cc:2468] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 1.
I20260202 15:07:09.985286 12072 leader_election.cc:304] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:09.985677 12298 tablet_bootstrap.cc:492] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6: No bootstrap required, opened a new log
I20260202 15:07:09.986052 12298 ts_tablet_manager.cc:1403] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260202 15:07:09.985836 12303 leader_election.cc:290] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:09.985975 12305 raft_consensus.cc:2804] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:09.986538 12305 raft_consensus.cc:697] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:09.987620 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12" is_pre_election: true
I20260202 15:07:09.987540 12305 consensus_queue.cc:237] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } }
I20260202 15:07:09.988530 12184 raft_consensus.cc:2468] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 0.
I20260202 15:07:09.989679 12298 raft_consensus.cc:359] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.991456 12298 raft_consensus.cc:385] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.991638 12300 raft_consensus.cc:359] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.992027 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6" is_pre_election: true
I20260202 15:07:09.993345 12298 raft_consensus.cc:740] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.993469 12300 raft_consensus.cc:385] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:09.993652 12109 raft_consensus.cc:2468] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 0.
I20260202 15:07:09.993124 12299 tablet_bootstrap.cc:492] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:09.993697 12300 raft_consensus.cc:740] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:09.990374 12221 leader_election.cc:304] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:09.993917 12298 consensus_queue.cc:260] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.994746 12303 raft_consensus.cc:2804] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:09.994573 12300 consensus_queue.cc:260] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.995116 12303 raft_consensus.cc:493] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:09.996155 12298 ts_tablet_manager.cc:1434] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260202 15:07:09.996852 12305 raft_consensus.cc:493] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:09.997078 12299 ts_tablet_manager.cc:1403] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.068s user 0.012s sys 0.010s
I20260202 15:07:09.997313 12305 raft_consensus.cc:515] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.996845 12300 ts_tablet_manager.cc:1434] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260202 15:07:09.999573 12299 raft_consensus.cc:359] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:09.999456 12004 catalog_manager.cc:5654] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 1 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.000141 12299 raft_consensus.cc:385] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:10.000104 12300 tablet_bootstrap.cc:492] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:10.000571 12299 raft_consensus.cc:740] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:10.001145 12305 leader_election.cc:290] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.001693 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" is_pre_election: true
I20260202 15:07:09.995481 12303 raft_consensus.cc:3060] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.002662 12257 raft_consensus.cc:2393] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93e91e2eb8834e7b8d844526093e59d6 in current term 1: Already voted for candidate 3270daa0942c4603b01ef4ddab75ec12 in this term.
I20260202 15:07:10.004281 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12" is_pre_election: true
I20260202 15:07:10.006578 12300 tablet_bootstrap.cc:654] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:10.010234 12072 leader_election.cc:304] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93e91e2eb8834e7b8d844526093e59d6; no voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12
I20260202 15:07:10.011339 12343 raft_consensus.cc:3060] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.004159 12299 consensus_queue.cc:260] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.013603 12299 ts_tablet_manager.cc:1434] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260202 15:07:10.014844 12299 tablet_bootstrap.cc:492] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:10.018296 12343 raft_consensus.cc:2749] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:10.008138 12305 raft_consensus.cc:493] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:10.009248 12303 raft_consensus.cc:515] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.022178 12299 tablet_bootstrap.cc:654] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:10.022892 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:10.023505 12183 raft_consensus.cc:3060] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.025657 12300 tablet_bootstrap.cc:492] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:10.026077 12300 ts_tablet_manager.cc:1403] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.001s
I20260202 15:07:10.028023 12299 tablet_bootstrap.cc:492] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:10.028308 12299 ts_tablet_manager.cc:1403] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260202 15:07:10.028645 12300 raft_consensus.cc:359] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.029342 12300 raft_consensus.cc:385] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:10.029371 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:10.029737 12300 raft_consensus.cc:740] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:10.030014 12109 raft_consensus.cc:3060] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.030457 12299 raft_consensus.cc:359] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.030580 12300 consensus_queue.cc:260] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.031212 12299 raft_consensus.cc:385] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:10.031538 12299 raft_consensus.cc:740] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:10.032486 12299 consensus_queue.cc:260] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.034899 12299 ts_tablet_manager.cc:1434] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260202 15:07:10.035054 12300 ts_tablet_manager.cc:1434] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260202 15:07:10.035804 12299 tablet_bootstrap.cc:492] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61: Bootstrap starting.
I20260202 15:07:10.035974 12300 tablet_bootstrap.cc:492] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12: Bootstrap starting.
I20260202 15:07:10.041417 12299 tablet_bootstrap.cc:654] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:10.042339 12303 leader_election.cc:290] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:10.045099 12109 raft_consensus.cc:2468] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 1.
I20260202 15:07:10.046227 12299 tablet_bootstrap.cc:492] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61: No bootstrap required, opened a new log
I20260202 15:07:10.046335 12223 leader_election.cc:304] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.046751 12299 ts_tablet_manager.cc:1403] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260202 15:07:10.047401 12303 raft_consensus.cc:2804] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:10.047801 12305 raft_consensus.cc:515] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.048592 12300 tablet_bootstrap.cc:654] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:10.048909 12303 raft_consensus.cc:697] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Becoming Leader. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:10.046777 12183 raft_consensus.cc:2468] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 1.
I20260202 15:07:10.049780 12303 consensus_queue.cc:237] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.050649 12299 raft_consensus.cc:359] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.051283 12299 raft_consensus.cc:385] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:10.051573 12299 raft_consensus.cc:740] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Initialized, Role: FOLLOWER
I20260202 15:07:10.052327 12299 consensus_queue.cc:260] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.052838 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12" is_pre_election: true
I20260202 15:07:10.053717 12183 raft_consensus.cc:2468] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 0.
I20260202 15:07:10.054742 12299 ts_tablet_manager.cc:1434] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260202 15:07:10.055023 12072 leader_election.cc:304] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.056041 12343 raft_consensus.cc:2804] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:10.056484 12343 raft_consensus.cc:493] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:10.056924 12343 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.057806 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" is_pre_election: true
I20260202 15:07:10.058415 12257 raft_consensus.cc:2468] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 0.
I20260202 15:07:10.059477 12300 tablet_bootstrap.cc:492] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12: No bootstrap required, opened a new log
I20260202 15:07:10.059576 12305 leader_election.cc:290] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.059924 12300 ts_tablet_manager.cc:1403] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.004s
I20260202 15:07:10.060250 12305 raft_consensus.cc:493] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:10.060671 12305 raft_consensus.cc:515] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.062368 12300 raft_consensus.cc:359] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.063179 12300 raft_consensus.cc:385] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:10.063482 12300 raft_consensus.cc:740] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Initialized, Role: FOLLOWER
I20260202 15:07:10.064373 12300 consensus_queue.cc:260] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.065200 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" is_pre_election: true
I20260202 15:07:10.065747 12257 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 0.
I20260202 15:07:10.066012 12343 raft_consensus.cc:515] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.066982 12075 leader_election.cc:304] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.068636 12300 ts_tablet_manager.cc:1434] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260202 15:07:10.069294 12343 leader_election.cc:290] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.069912 12343 raft_consensus.cc:2804] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:10.070225 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.070268 12343 raft_consensus.cc:493] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:10.070849 12257 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.070878 12343 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.070989 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:10.071668 12184 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.063468 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12" is_pre_election: true
I20260202 15:07:10.074049 12183 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 0.
I20260202 15:07:10.073242 12318 raft_consensus.cc:493] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:10.075536 12318 raft_consensus.cc:515] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.077527 12318 leader_election.cc:290] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.079013 12184 raft_consensus.cc:2468] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 1.
I20260202 15:07:10.079558 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" is_pre_election: true
I20260202 15:07:10.080173 12258 raft_consensus.cc:2468] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 0.
I20260202 15:07:10.080430 12343 raft_consensus.cc:515] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.081924 12150 leader_election.cc:304] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:10.083027 12072 leader_election.cc:304] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.084771 12257 raft_consensus.cc:2468] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 1.
I20260202 15:07:10.086304 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.087069 12258 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.085181 12343 leader_election.cc:290] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.087905 12343 raft_consensus.cc:2804] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:10.086206 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:10.088387 12343 raft_consensus.cc:697] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:10.088073 12318 raft_consensus.cc:2804] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:10.088511 12305 leader_election.cc:290] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.089547 12184 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.090904 12318 raft_consensus.cc:493] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:10.092351 12318 raft_consensus.cc:3060] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.091851 12305 raft_consensus.cc:493] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:10.095048 12305 raft_consensus.cc:515] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.098779 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05b935b29d684295a2d3587ff36f921e" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12" is_pre_election: true
I20260202 15:07:10.099575 12183 raft_consensus.cc:2468] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 0.
I20260202 15:07:10.099686 12258 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 1.
I20260202 15:07:10.100871 12072 leader_election.cc:304] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.100251 12343 consensus_queue.cc:237] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.101619 12075 leader_election.cc:304] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.102171 12353 raft_consensus.cc:2804] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:10.102824 12353 raft_consensus.cc:493] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:10.103211 12353 raft_consensus.cc:3060] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.109035 12353 raft_consensus.cc:515] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.111709 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05b935b29d684295a2d3587ff36f921e" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:10.112246 12183 raft_consensus.cc:3060] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.112366 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05b935b29d684295a2d3587ff36f921e" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.112826 12353 leader_election.cc:290] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.113358 12353 raft_consensus.cc:2804] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:10.113771 12353 raft_consensus.cc:697] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:10.114548 12353 consensus_queue.cc:237] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.116843 12258 raft_consensus.cc:3060] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.121498 12183 raft_consensus.cc:2468] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 1.
I20260202 15:07:10.124272 12305 leader_election.cc:290] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.124876 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05b935b29d684295a2d3587ff36f921e" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" is_pre_election: true
I20260202 15:07:10.124840 12258 raft_consensus.cc:2468] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 1.
I20260202 15:07:10.128337 12072 leader_election.cc:304] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.129415 12354 raft_consensus.cc:2804] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:10.133970 12004 catalog_manager.cc:5654] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:10.136603 12184 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 1.
I20260202 15:07:10.149092 12007 catalog_manager.cc:5654] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 1 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.159274 12318 raft_consensus.cc:515] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.161438 12318 leader_election.cc:290] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.162195 12354 raft_consensus.cc:697] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:10.162937 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.163090 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6" is_pre_election: true
I20260202 15:07:10.163619 12258 raft_consensus.cc:3060] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.163779 12109 raft_consensus.cc:2468] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 0.
I20260202 15:07:10.163519 12354 consensus_queue.cc:237] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.166574 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:10.167208 12109 raft_consensus.cc:3060] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.170003 12258 raft_consensus.cc:2468] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 1.
I20260202 15:07:10.170949 12150 leader_election.cc:304] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:10.171641 12318 raft_consensus.cc:2804] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:10.172075 12318 raft_consensus.cc:697] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:10.172245 12007 catalog_manager.cc:5654] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 1 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.172778 12318 consensus_queue.cc:237] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.173811 12007 catalog_manager.cc:5654] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 1 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.175462 12109 raft_consensus.cc:2468] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 1.
I20260202 15:07:10.180881 12004 catalog_manager.cc:5654] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "3270daa0942c4603b01ef4ddab75ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.185392 12184 raft_consensus.cc:3060] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.191411 12303 consensus_queue.cc:1048] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.202081 12363 consensus_queue.cc:1048] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:10.211687 12318 raft_consensus.cc:493] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:10.212112 12318 raft_consensus.cc:515] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.214042 12318 leader_election.cc:290] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.214576 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6" is_pre_election: true
I20260202 15:07:10.215134 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" is_pre_election: true
I20260202 15:07:10.215145 12109 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 0.
I20260202 15:07:10.215713 12258 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 0.
I20260202 15:07:10.216223 12149 leader_election.cc:304] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.217015 12318 raft_consensus.cc:2804] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:10.217300 12318 raft_consensus.cc:493] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:10.217540 12318 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.222211 12318 raft_consensus.cc:515] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.223687 12318 leader_election.cc:290] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.224295 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:10.224519 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.224807 12109 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.225029 12258 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.229053 12109 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 1.
I20260202 15:07:10.229200 12258 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 1.
I20260202 15:07:10.229904 12149 leader_election.cc:304] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.230571 12318 raft_consensus.cc:2804] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:10.231019 12318 raft_consensus.cc:697] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:10.231554 12318 consensus_queue.cc:237] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.237931 12004 catalog_manager.cc:5654] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "3270daa0942c4603b01ef4ddab75ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.253620 12318 consensus_queue.cc:1048] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.259282 12109 raft_consensus.cc:3060] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:10.267254 12329 consensus_queue.cc:1048] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:10.365935 12329 consensus_queue.cc:1048] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.372632 12184 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e"
dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
mode: GRACEFUL
new_leader_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:40234
I20260202 15:07:10.374259 12184 raft_consensus.cc:606] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Received request to transfer leadership to TS 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:10.376636 12308 consensus_queue.cc:1048] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.386063 12308 raft_consensus.cc:993] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12: : Instructing follower 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:10.386444 12308 raft_consensus.cc:1081] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Signalling peer 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:10.388116 12109 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:60068
I20260202 15:07:10.388679 12109 raft_consensus.cc:493] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:10.389019 12109 raft_consensus.cc:3060] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.392987 12258 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd"
dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
mode: GRACEFUL
new_leader_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:46906
I20260202 15:07:10.393358 12258 raft_consensus.cc:606] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Received request to transfer leadership to TS 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:10.394275 12109 raft_consensus.cc:515] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.396571 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.396576 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0ebeba16c704c3eb18cacfbfc74d95e" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:10.397251 12258 raft_consensus.cc:3060] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.397312 12183 raft_consensus.cc:3055] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:07:10.397681 12183 raft_consensus.cc:740] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:10.397828 12184 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314"
dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
mode: GRACEFUL
new_leader_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:40234
I20260202 15:07:10.397743 12354 consensus_queue.cc:1048] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.398361 12184 raft_consensus.cc:606] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Received request to transfer leadership to TS 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:10.398517 12183 consensus_queue.cc:260] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.399475 12183 raft_consensus.cc:3060] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.403342 12258 raft_consensus.cc:2468] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 2.
I20260202 15:07:10.404369 12075 leader_election.cc:304] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.405223 12354 raft_consensus.cc:2804] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:07:10.405659 12354 raft_consensus.cc:697] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:10.406440 12354 consensus_queue.cc:237] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.411986 12343 consensus_queue.cc:1048] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.425822 12183 raft_consensus.cc:2468] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 2.
I20260202 15:07:10.429255 12109 leader_election.cc:290] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 2 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.429661 12108 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
mode: GRACEFUL
new_leader_uuid: "3270daa0942c4603b01ef4ddab75ec12"
from {username='slave'} at 127.0.0.1:60070
I20260202 15:07:10.430213 12108 raft_consensus.cc:606] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Received request to transfer leadership to TS 3270daa0942c4603b01ef4ddab75ec12
I20260202 15:07:10.434881 12257 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7"
dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
mode: GRACEFUL
new_leader_uuid: "3270daa0942c4603b01ef4ddab75ec12"
from {username='slave'} at 127.0.0.1:46906
I20260202 15:07:10.435321 12257 raft_consensus.cc:606] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Received request to transfer leadership to TS 3270daa0942c4603b01ef4ddab75ec12
I20260202 15:07:10.434957 12006 catalog_manager.cc:5654] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 1 to 2, leader changed from 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194) to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 2 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.445029 12108 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "05b935b29d684295a2d3587ff36f921e"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
mode: GRACEFUL
new_leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:60070
I20260202 15:07:10.445556 12108 raft_consensus.cc:606] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Received request to transfer leadership to TS 0776af4ab68c4c79a6110f2f2a81dc61
I20260202 15:07:10.489506 12303 consensus_queue.cc:1048] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.495373 12363 raft_consensus.cc:993] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61: : Instructing follower 3270daa0942c4603b01ef4ddab75ec12 to start an election
I20260202 15:07:10.495867 12322 raft_consensus.cc:1081] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Signalling peer 3270daa0942c4603b01ef4ddab75ec12 to start an election
I20260202 15:07:10.497438 12182 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7"
dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
from {username='slave'} at 127.0.0.1:40210
I20260202 15:07:10.497951 12182 raft_consensus.cc:493] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:10.498265 12182 raft_consensus.cc:3060] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.499761 12358 consensus_queue.cc:1048] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.501392 12303 consensus_queue.cc:1048] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.505807 12182 raft_consensus.cc:515] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.509403 12183 raft_consensus.cc:1240] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Rejecting Update request from peer 0776af4ab68c4c79a6110f2f2a81dc61 for earlier term 1. Current term is 2. Ops: []
I20260202 15:07:10.509811 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:10.510324 12109 raft_consensus.cc:3060] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.512573 12182 leader_election.cc:290] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 2 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.510324 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.514479 12257 raft_consensus.cc:3055] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:07:10.514844 12257 raft_consensus.cc:740] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:10.515486 12257 consensus_queue.cc:260] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.516376 12257 raft_consensus.cc:3060] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.517607 12109 raft_consensus.cc:2468] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 2.
I20260202 15:07:10.520650 12149 leader_election.cc:304] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.521384 12318 raft_consensus.cc:2804] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:07:10.521631 12353 consensus_queue.cc:1048] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.521922 12318 raft_consensus.cc:697] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Becoming Leader. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:10.522786 12318 consensus_queue.cc:237] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.524308 12257 raft_consensus.cc:2468] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 2.
I20260202 15:07:10.534474 12006 catalog_manager.cc:5654] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 reported cstate change: term changed from 1 to 2, leader changed from 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195) to 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "3270daa0942c4603b01ef4ddab75ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.545120 12353 consensus_queue.cc:1048] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.554294 12358 consensus_queue.cc:1048] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.561266 12343 raft_consensus.cc:993] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6: : Instructing follower 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:10.561637 12358 raft_consensus.cc:1081] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Signalling peer 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:10.562391 12343 consensus_queue.cc:1048] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.562968 12258 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "05b935b29d684295a2d3587ff36f921e"
dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:46892
I20260202 15:07:10.563393 12258 raft_consensus.cc:493] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:10.563699 12258 raft_consensus.cc:3060] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.568521 12343 raft_consensus.cc:993] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6: : Instructing follower 3270daa0942c4603b01ef4ddab75ec12 to start an election
I20260202 15:07:10.568890 12343 raft_consensus.cc:1081] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Signalling peer 3270daa0942c4603b01ef4ddab75ec12 to start an election
I20260202 15:07:10.570574 12183 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1"
dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
from {username='slave'} at 127.0.0.1:40224
I20260202 15:07:10.571450 12183 raft_consensus.cc:493] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:10.571772 12183 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.574571 12358 consensus_queue.cc:1048] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.570286 12258 raft_consensus.cc:515] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.576835 12183 raft_consensus.cc:515] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.578507 12108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05b935b29d684295a2d3587ff36f921e" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:10.579034 12108 raft_consensus.cc:3055] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:07:10.579106 12258 leader_election.cc:290] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 2 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:10.579389 12108 raft_consensus.cc:740] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:10.580087 12256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.580049 12108 consensus_queue.cc:260] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.581220 12108 raft_consensus.cc:3060] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.581004 12183 leader_election.cc:290] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 2 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.583083 12109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:10.583596 12109 raft_consensus.cc:3055] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:07:10.583813 12318 consensus_queue.cc:1048] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.583992 12109 raft_consensus.cc:740] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:10.584666 12109 consensus_queue.cc:260] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.585462 12109 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.579058 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05b935b29d684295a2d3587ff36f921e" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:10.587625 12184 raft_consensus.cc:3060] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.594386 12318 consensus_queue.cc:1048] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.598992 12184 raft_consensus.cc:2468] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 2.
I20260202 15:07:10.603673 12109 raft_consensus.cc:2468] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 2.
I20260202 15:07:10.604763 12221 leader_election.cc:304] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:10.604892 12149 leader_election.cc:304] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters: 0776af4ab68c4c79a6110f2f2a81dc61
I20260202 15:07:10.605365 12303 raft_consensus.cc:2804] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:07:10.605749 12303 raft_consensus.cc:697] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 LEADER]: Becoming Leader. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:10.605886 12318 raft_consensus.cc:2804] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:07:10.606316 12318 raft_consensus.cc:697] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Becoming Leader. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:10.606411 12303 consensus_queue.cc:237] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.607115 12318 consensus_queue.cc:237] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.603845 12108 raft_consensus.cc:2468] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 2.
I20260202 15:07:10.613721 12004 catalog_manager.cc:5654] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 reported cstate change: term changed from 1 to 2, leader changed from 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193) to 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "3270daa0942c4603b01ef4ddab75ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.616914 12006 catalog_manager.cc:5654] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 reported cstate change: term changed from 1 to 2, leader changed from 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193) to 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:10.793224 12322 raft_consensus.cc:993] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61: : Instructing follower 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:10.793629 12303 raft_consensus.cc:1081] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Signalling peer 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:10.794952 12108 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:60062
I20260202 15:07:10.795399 12108 raft_consensus.cc:493] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:10.795698 12108 raft_consensus.cc:3060] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.800109 12108 raft_consensus.cc:515] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.802028 12108 leader_election.cc:290] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 2 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.802723 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:10.803287 12184 raft_consensus.cc:3060] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.803251 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.804200 12257 raft_consensus.cc:3055] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:07:10.804083 12318 consensus_queue.cc:1048] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.804502 12257 raft_consensus.cc:740] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:10.805202 12257 consensus_queue.cc:260] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.806329 12257 raft_consensus.cc:3060] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.809268 12184 raft_consensus.cc:2468] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 2.
I20260202 15:07:10.810858 12308 raft_consensus.cc:993] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12: : Instructing follower 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:10.811339 12308 raft_consensus.cc:1081] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Signalling peer 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:10.810362 12072 leader_election.cc:304] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.812026 12358 raft_consensus.cc:2804] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:07:10.812446 12358 raft_consensus.cc:697] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:10.812687 12257 raft_consensus.cc:2468] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 2.
I20260202 15:07:10.813103 12358 consensus_queue.cc:237] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.815171 12108 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:60068
I20260202 15:07:10.816558 12108 raft_consensus.cc:493] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:10.816987 12108 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.818055 12308 consensus_queue.cc:1048] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:10.821395 12004 catalog_manager.cc:5654] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 1 to 2, leader changed from 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195) to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 2 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.823211 12108 raft_consensus.cc:515] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.825405 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:10.825855 12184 raft_consensus.cc:3055] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:07:10.826160 12184 raft_consensus.cc:740] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:10.826884 12184 consensus_queue.cc:260] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.827943 12184 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.824981 12109 raft_consensus.cc:1240] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Rejecting Update request from peer 3270daa0942c4603b01ef4ddab75ec12 for earlier term 1. Current term is 2. Ops: []
I20260202 15:07:10.831524 12108 leader_election.cc:290] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 2 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:10.834610 12184 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 2.
I20260202 15:07:10.835433 12257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:10.835539 12072 leader_election.cc:304] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:10.835824 12257 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:10.836229 12358 raft_consensus.cc:2804] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:07:10.836651 12358 raft_consensus.cc:697] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:10.837474 12358 consensus_queue.cc:237] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:10.842206 12257 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 2.
I20260202 15:07:10.844038 12006 catalog_manager.cc:5654] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 1 to 2, leader changed from 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194) to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 2 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:10.877771 12257 raft_consensus.cc:1275] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:10.878961 12358 consensus_queue.cc:1048] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:10.885968 12184 raft_consensus.cc:1275] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:10.887698 12358 consensus_queue.cc:1048] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:10.999818 12108 raft_consensus.cc:1275] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Refusing update from remote peer 3270daa0942c4603b01ef4ddab75ec12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.001171 12308 consensus_queue.cc:1048] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:11.008201 12257 raft_consensus.cc:1275] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Refusing update from remote peer 3270daa0942c4603b01ef4ddab75ec12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.010021 12318 consensus_queue.cc:1048] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:11.020339 12108 raft_consensus.cc:1275] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.021690 12303 consensus_queue.cc:1048] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:11.029443 12184 raft_consensus.cc:1275] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.030974 12363 consensus_queue.cc:1048] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:11.036631 12257 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:11.043681 12257 raft_consensus.cc:1275] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Refusing update from remote peer 3270daa0942c4603b01ef4ddab75ec12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.045029 12318 consensus_queue.cc:1048] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:11.052570 12108 raft_consensus.cc:1275] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Refusing update from remote peer 3270daa0942c4603b01ef4ddab75ec12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.053956 12318 consensus_queue.cc:1048] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:11.236788 12184 raft_consensus.cc:1275] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.237953 12305 consensus_queue.cc:1048] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:11.244170 12257 raft_consensus.cc:1275] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.245278 12305 consensus_queue.cc:1048] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:11.267661 12257 raft_consensus.cc:1275] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.268945 12305 consensus_queue.cc:1048] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:11.271417 12184 raft_consensus.cc:1275] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:11.272881 12358 consensus_queue.cc:1048] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:13.504585 11451 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20260202 15:07:13.505424 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:13.539479 12108 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
mode: GRACEFUL
new_leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:60072
I20260202 15:07:13.540076 12108 raft_consensus.cc:606] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Received request to transfer leadership to TS 0776af4ab68c4c79a6110f2f2a81dc61
I20260202 15:07:13.543010 12108 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
mode: GRACEFUL
new_leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:60072
I20260202 15:07:13.543512 12108 raft_consensus.cc:606] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Received request to transfer leadership to TS 0776af4ab68c4c79a6110f2f2a81dc61
I20260202 15:07:13.544931 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260202 15:07:13.545418 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:13.819137 12343 raft_consensus.cc:993] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6: : Instructing follower 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:13.819694 12343 raft_consensus.cc:1081] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Signalling peer 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:13.821283 12257 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace"
dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:46892
I20260202 15:07:13.821856 12257 raft_consensus.cc:493] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:13.822245 12257 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:13.831161 12257 raft_consensus.cc:515] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:13.833890 12257 leader_election.cc:290] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 2 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:13.837656 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:13.837879 12108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:13.838315 12184 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:13.838508 12108 raft_consensus.cc:3055] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:07:13.838927 12108 raft_consensus.cc:740] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:13.839730 12108 consensus_queue.cc:260] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:13.840979 12108 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:13.843990 12184 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 2.
I20260202 15:07:13.845245 12221 leader_election.cc:304] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:13.845959 12322 raft_consensus.cc:2804] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:07:13.846153 12108 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 2.
I20260202 15:07:13.849452 12322 raft_consensus.cc:697] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 LEADER]: Becoming Leader. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:13.857309 12322 consensus_queue.cc:237] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:13.864354 12004 catalog_manager.cc:5654] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 reported cstate change: term changed from 1 to 2, leader changed from 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193) to 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:13.881363 12343 raft_consensus.cc:993] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6: : Instructing follower 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:13.881933 12343 raft_consensus.cc:1081] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Signalling peer 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:13.883595 12257 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314"
dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:46892
I20260202 15:07:13.884142 12257 raft_consensus.cc:493] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:13.884516 12257 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:13.893204 12257 raft_consensus.cc:515] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:13.897310 12257 leader_election.cc:290] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 3 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:13.898011 12108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:13.898617 12108 raft_consensus.cc:3055] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:07:13.898602 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:13.899132 12108 raft_consensus.cc:740] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:13.899287 12184 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:13.899955 12108 consensus_queue.cc:260] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:13.901261 12108 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:13.906740 12184 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 3.
I20260202 15:07:13.908437 12108 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 3.
I20260202 15:07:13.908391 12221 leader_election.cc:304] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:13.909649 12322 raft_consensus.cc:2804] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:07:13.910619 12322 raft_consensus.cc:697] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 LEADER]: Becoming Leader. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:13.911427 12322 consensus_queue.cc:237] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:13.920220 12004 catalog_manager.cc:5654] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 reported cstate change: term changed from 2 to 3, leader changed from 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193) to 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:14.350373 12109 raft_consensus.cc:1275] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:14.351734 12322 consensus_queue.cc:1048] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:14.379837 12184 raft_consensus.cc:1275] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:14.382421 12405 consensus_queue.cc:1048] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:07:14.384159 12183 raft_consensus.cc:1275] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:14.386513 12405 consensus_queue.cc:1048] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:07:14.402170 12109 raft_consensus.cc:1275] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:14.403729 12405 consensus_queue.cc:1048] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:14.546428 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:14.559885 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:14.560613 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:15.561997 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:15.569387 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:15.569948 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:16.573177 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:16.581727 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:16.582238 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:17.583302 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:17.590292 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:17.591315 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:18.594192 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:18.602535 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:18.603019 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:19.604167 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:19.610212 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:19.611099 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260202 15:07:21.399034 12038 debug-util.cc:398] Leaking SignalData structure 0x7b0800052000 after lost signal to thread 11454
W20260202 15:07:21.399955 12038 debug-util.cc:398] Leaking SignalData structure 0x7b08000526e0 after lost signal to thread 12041
W20260202 15:07:21.400794 12038 debug-util.cc:398] Leaking SignalData structure 0x7b08000dd560 after lost signal to thread 12133
W20260202 15:07:21.401562 12038 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc920 after lost signal to thread 12208
W20260202 15:07:21.402348 12038 debug-util.cc:398] Leaking SignalData structure 0x7b080013f280 after lost signal to thread 12282
I20260202 15:07:21.539536 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:21.546676 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:21.547178 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:22.553275 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:22.560842 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:22.561345 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:23.568498 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:23.966580 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:24.111997 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:25.165318 11451 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20260202 15:07:25.178576 12255 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace"
dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
mode: GRACEFUL
new_leader_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:46906
I20260202 15:07:25.179781 12255 raft_consensus.cc:606] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 LEADER]: Received request to transfer leadership to TS 93e91e2eb8834e7b8d844526093e59d6
W20260202 15:07:25.183781 12279 debug-util.cc:398] Leaking SignalData structure 0x7b08000ab460 after lost signal to thread 11454
W20260202 15:07:25.184798 12279 debug-util.cc:398] Leaking SignalData structure 0x7b08000abb80 after lost signal to thread 12041
W20260202 15:07:25.185808 12279 debug-util.cc:398] Leaking SignalData structure 0x7b080011b0a0 after lost signal to thread 12133
W20260202 15:07:25.186731 12279 debug-util.cc:398] Leaking SignalData structure 0x7b08001951e0 after lost signal to thread 12208
W20260202 15:07:25.187673 12279 debug-util.cc:398] Leaking SignalData structure 0x7b08000c4520 after lost signal to thread 12282
I20260202 15:07:25.196873 12255 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314"
dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
mode: GRACEFUL
new_leader_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:46906
I20260202 15:07:25.197568 12255 raft_consensus.cc:606] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 LEADER]: Received request to transfer leadership to TS 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:25.204288 12182 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1"
dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
mode: GRACEFUL
new_leader_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:40234
I20260202 15:07:25.205019 12182 raft_consensus.cc:606] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Received request to transfer leadership to TS 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:25.213675 12182 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7"
dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
mode: GRACEFUL
new_leader_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:40234
I20260202 15:07:25.214511 12182 raft_consensus.cc:606] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Received request to transfer leadership to TS 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:25.223806 12182 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b"
dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
mode: GRACEFUL
new_leader_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:40234
I20260202 15:07:25.224550 12182 raft_consensus.cc:606] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Received request to transfer leadership to TS 93e91e2eb8834e7b8d844526093e59d6
I20260202 15:07:25.508050 12435 raft_consensus.cc:993] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12: : Instructing follower 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.509084 12444 raft_consensus.cc:1081] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Signalling peer 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.511858 12108 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:60068
I20260202 15:07:25.512732 12108 raft_consensus.cc:493] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:25.513360 12108 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:25.527052 12108 raft_consensus.cc:515] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.529733 12445 raft_consensus.cc:993] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61: : Instructing follower 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.530992 12445 raft_consensus.cc:1081] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 LEADER]: Signalling peer 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.533887 12107 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:60062
I20260202 15:07:25.534619 12107 raft_consensus.cc:493] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:25.535650 12107 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:07:25.535547 12182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:25.536501 12182 raft_consensus.cc:3055] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:07:25.536803 12182 raft_consensus.cc:740] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:25.537652 12182 consensus_queue.cc:260] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.533121 12444 raft_consensus.cc:993] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12: : Instructing follower 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.539300 12444 raft_consensus.cc:1081] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Signalling peer 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.535545 12255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:25.540508 12255 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:25.541860 12106 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:60068
I20260202 15:07:25.542483 12106 raft_consensus.cc:493] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:25.542914 12106 raft_consensus.cc:3060] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:25.539253 12182 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:25.548666 12107 raft_consensus.cc:515] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.550868 12107 leader_election.cc:290] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 4 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:25.551877 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:25.552026 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f27edb4a944789ade0124dbf15d314" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:25.552502 12184 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:07:25.552549 12258 raft_consensus.cc:3055] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:07:25.552937 12258 raft_consensus.cc:740] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:25.553946 12258 consensus_queue.cc:260] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.555637 12258 raft_consensus.cc:3060] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:07:25.560521 12184 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 4.
I20260202 15:07:25.561978 12072 leader_election.cc:304] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:25.563129 12438 raft_consensus.cc:2804] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:07:25.562848 12106 raft_consensus.cc:515] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.564324 12438 raft_consensus.cc:697] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 4 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:25.565408 12438 consensus_queue.cc:237] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.566200 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:25.566669 12441 raft_consensus.cc:993] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61: : Instructing follower 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.566941 12183 raft_consensus.cc:3055] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:07:25.567355 12183 raft_consensus.cc:740] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:25.568075 12445 raft_consensus.cc:1081] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 LEADER]: Signalling peer 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.568372 12183 consensus_queue.cc:260] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.569957 12183 raft_consensus.cc:3060] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:25.572506 12255 raft_consensus.cc:2468] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 3.
I20260202 15:07:25.573145 12106 leader_election.cc:290] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 3 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:25.573186 12256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "763ffb7935a149728615aec0b0f953f7" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:25.573901 12256 raft_consensus.cc:3060] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:25.574044 12075 leader_election.cc:304] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:25.582178 12258 raft_consensus.cc:2468] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 4.
I20260202 15:07:25.569958 12107 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:60062
I20260202 15:07:25.582787 12108 leader_election.cc:290] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 3 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:25.582914 12107 raft_consensus.cc:493] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:25.585490 12107 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:25.592773 12182 raft_consensus.cc:2468] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 3.
I20260202 15:07:25.593534 12448 raft_consensus.cc:2804] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:07:25.595176 12107 raft_consensus.cc:515] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.600894 12256 raft_consensus.cc:2468] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 3.
I20260202 15:07:25.601794 12107 leader_election.cc:290] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 3 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:25.597329 12004 catalog_manager.cc:5654] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 3 to 4, leader changed from 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195) to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 4 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:25.601002 12184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:25.607020 12184 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:25.601267 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:25.607949 12258 raft_consensus.cc:3055] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:07:25.608362 12258 raft_consensus.cc:740] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:25.602448 12075 leader_election.cc:304] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:25.595629 12448 raft_consensus.cc:697] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:25.611362 12438 raft_consensus.cc:2804] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:07:25.620990 12184 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 3.
I20260202 15:07:25.620864 12448 consensus_queue.cc:237] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.610421 12258 consensus_queue.cc:260] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.628330 12258 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:25.610829 12183 raft_consensus.cc:2468] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 3.
I20260202 15:07:25.634603 12438 raft_consensus.cc:697] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:25.636546 12072 leader_election.cc:304] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:25.644502 12444 raft_consensus.cc:993] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12: : Instructing follower 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.645294 12444 raft_consensus.cc:1081] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Signalling peer 93e91e2eb8834e7b8d844526093e59d6 to start an election
I20260202 15:07:25.640424 12438 consensus_queue.cc:237] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.654268 12453 raft_consensus.cc:2804] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:07:25.657078 12107 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
from {username='slave'} at 127.0.0.1:60068
I20260202 15:07:25.658406 12453 raft_consensus.cc:697] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:25.659080 12107 raft_consensus.cc:493] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:25.659703 12107 raft_consensus.cc:3060] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:25.660012 12453 consensus_queue.cc:237] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.674151 12107 raft_consensus.cc:515] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.679869 12183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:25.680565 12183 raft_consensus.cc:3055] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:07:25.680967 12183 raft_consensus.cc:740] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:25.682133 12183 consensus_queue.cc:260] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.683880 12183 raft_consensus.cc:3060] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:25.700220 12258 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 3.
I20260202 15:07:25.702932 12107 leader_election.cc:290] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 2 election: Requested vote from peers 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:25.702865 12183 raft_consensus.cc:2468] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 2.
I20260202 15:07:25.705899 12072 leader_election.cc:304] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3270daa0942c4603b01ef4ddab75ec12, 93e91e2eb8834e7b8d844526093e59d6; no voters:
I20260202 15:07:25.707471 12453 raft_consensus.cc:2804] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:07:25.708190 12453 raft_consensus.cc:697] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Becoming Leader. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:25.713162 12453 consensus_queue.cc:237] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:25.720396 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "93e91e2eb8834e7b8d844526093e59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:25.721414 12258 raft_consensus.cc:3060] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:25.748466 12258 raft_consensus.cc:2468] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e91e2eb8834e7b8d844526093e59d6 in term 2.
I20260202 15:07:25.781023 12004 catalog_manager.cc:5654] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 2 to 3, leader changed from 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195) to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 3 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:25.782500 12004 catalog_manager.cc:5654] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 2 to 3, leader changed from 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194) to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 3 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:25.783754 12004 catalog_manager.cc:5654] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 1 to 2, leader changed from 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194) to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 2 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:25.784929 12004 catalog_manager.cc:5654] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 reported cstate change: term changed from 2 to 3, leader changed from 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194) to 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193). New cstate: current_term: 3 leader_uuid: "93e91e2eb8834e7b8d844526093e59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:26.035373 12258 raft_consensus.cc:1275] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:26.039670 12453 consensus_queue.cc:1048] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:26.059989 12183 raft_consensus.cc:1275] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:26.060395 12256 raft_consensus.cc:1275] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:07:26.062337 12448 consensus_queue.cc:1048] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:07:26.063463 12453 consensus_queue.cc:1048] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:26.083662 12183 raft_consensus.cc:1275] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:07:26.084774 12255 raft_consensus.cc:1275] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:26.085934 12453 consensus_queue.cc:1048] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260202 15:07:26.086562 12438 consensus_queue.cc:1048] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:07:26.105463 12182 raft_consensus.cc:1275] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:26.114243 12438 consensus_queue.cc:1048] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:07:26.183890 12182 raft_consensus.cc:1275] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:26.199615 12438 consensus_queue.cc:1048] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:07:26.188704 12181 raft_consensus.cc:1275] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:26.214082 12471 consensus_queue.cc:1048] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:07:26.188078 12256 raft_consensus.cc:1275] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:26.246162 12453 consensus_queue.cc:1048] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260202 15:07:26.347499 12223 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40833, user_credentials={real_user=slave}} blocked reactor thread for 60177.6us
I20260202 15:07:26.367136 12258 raft_consensus.cc:1275] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Refusing update from remote peer 93e91e2eb8834e7b8d844526093e59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:26.384640 12472 consensus_queue.cc:1048] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:07:28.264235 11451 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20260202 15:07:28.264967 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:28.271118 12107 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
mode: GRACEFUL
new_leader_uuid: "3270daa0942c4603b01ef4ddab75ec12"
from {username='slave'} at 127.0.0.1:60072
I20260202 15:07:28.271865 12107 raft_consensus.cc:606] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Received request to transfer leadership to TS 3270daa0942c4603b01ef4ddab75ec12
I20260202 15:07:28.273857 12107 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
mode: GRACEFUL
new_leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:60072
I20260202 15:07:28.274381 12107 raft_consensus.cc:606] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Received request to transfer leadership to TS 0776af4ab68c4c79a6110f2f2a81dc61
I20260202 15:07:28.276388 12107 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
mode: GRACEFUL
new_leader_uuid: "3270daa0942c4603b01ef4ddab75ec12"
from {username='slave'} at 127.0.0.1:60072
I20260202 15:07:28.276711 12107 raft_consensus.cc:606] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Received request to transfer leadership to TS 3270daa0942c4603b01ef4ddab75ec12
I20260202 15:07:28.278481 12107 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1"
dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
mode: GRACEFUL
new_leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:60072
I20260202 15:07:28.278918 12107 raft_consensus.cc:606] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Received request to transfer leadership to TS 0776af4ab68c4c79a6110f2f2a81dc61
I20260202 15:07:28.279784 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260202 15:07:28.280143 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:28.431629 12483 raft_consensus.cc:993] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6: : Instructing follower 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:28.432072 12471 raft_consensus.cc:1081] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Signalling peer 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:28.433157 12258 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b"
dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:46892
I20260202 15:07:28.433506 12258 raft_consensus.cc:493] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:28.433744 12258 raft_consensus.cc:3060] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:28.438485 12258 raft_consensus.cc:515] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.440227 12258 leader_election.cc:290] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 3 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:28.440847 12107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:28.441309 12181 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58af84d12bfe4f359253736f8bbdce4b" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:28.441538 12107 raft_consensus.cc:3055] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:07:28.441780 12181 raft_consensus.cc:3060] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:28.441825 12107 raft_consensus.cc:740] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:28.442586 12107 consensus_queue.cc:260] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.443989 12107 raft_consensus.cc:3060] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:28.448575 12181 raft_consensus.cc:2468] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 3.
I20260202 15:07:28.449879 12221 leader_election.cc:304] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:28.450664 12479 raft_consensus.cc:2804] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:07:28.450793 12107 raft_consensus.cc:2468] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 3.
I20260202 15:07:28.451692 12479 raft_consensus.cc:697] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 LEADER]: Becoming Leader. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:28.452417 12479 consensus_queue.cc:237] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.458884 12007 catalog_manager.cc:5654] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 reported cstate change: term changed from 2 to 3, leader changed from 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193) to 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:28.491557 12483 raft_consensus.cc:993] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6: : Instructing follower 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:28.491982 12471 raft_consensus.cc:1081] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Signalling peer 0776af4ab68c4c79a6110f2f2a81dc61 to start an election
I20260202 15:07:28.493052 12258 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1"
dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
from {username='slave'} at 127.0.0.1:46892
I20260202 15:07:28.493427 12258 raft_consensus.cc:493] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:28.493680 12258 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:07:28.497349 12258 raft_consensus.cc:515] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.498481 12258 leader_election.cc:290] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 4 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227)
I20260202 15:07:28.499109 12107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:28.499346 12181 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c74740e8b5ef4ff8b61d37b275dd4dc1" candidate_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
I20260202 15:07:28.499681 12107 raft_consensus.cc:3055] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:07:28.499843 12181 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:07:28.499970 12107 raft_consensus.cc:740] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:28.500685 12107 consensus_queue.cc:260] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.501816 12107 raft_consensus.cc:3060] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:07:28.506291 12181 raft_consensus.cc:2468] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 4.
I20260202 15:07:28.507166 12221 leader_election.cc:304] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:28.507737 12479 raft_consensus.cc:2804] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:07:28.507908 12107 raft_consensus.cc:2468] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776af4ab68c4c79a6110f2f2a81dc61 in term 4.
I20260202 15:07:28.508193 12479 raft_consensus.cc:697] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 LEADER]: Becoming Leader. State: Replica: 0776af4ab68c4c79a6110f2f2a81dc61, State: Running, Role: LEADER
I20260202 15:07:28.509096 12479 consensus_queue.cc:237] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.515411 12007 catalog_manager.cc:5654] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 reported cstate change: term changed from 3 to 4, leader changed from 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193) to 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195). New cstate: current_term: 4 leader_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:28.550638 12483 raft_consensus.cc:993] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6: : Instructing follower 3270daa0942c4603b01ef4ddab75ec12 to start an election
I20260202 15:07:28.551175 12481 raft_consensus.cc:1081] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Signalling peer 3270daa0942c4603b01ef4ddab75ec12 to start an election
I20260202 15:07:28.552455 12181 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace"
dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
from {username='slave'} at 127.0.0.1:40224
I20260202 15:07:28.552946 12181 raft_consensus.cc:493] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:28.553290 12181 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:07:28.556921 12181 raft_consensus.cc:515] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.558629 12181 leader_election.cc:290] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 4 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:28.559353 12107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:28.559574 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68bb2b68bcfb46aca45082c97b6b4ace" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:28.559851 12107 raft_consensus.cc:3055] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:07:28.560070 12258 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:07:28.560164 12107 raft_consensus.cc:740] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:28.560817 12107 consensus_queue.cc:260] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.561682 12107 raft_consensus.cc:3060] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:07:28.563640 12258 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 4.
I20260202 15:07:28.564601 12150 leader_election.cc:304] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:28.565312 12484 raft_consensus.cc:2804] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:07:28.565798 12107 raft_consensus.cc:2468] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 4.
I20260202 15:07:28.566341 12484 raft_consensus.cc:697] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 4 LEADER]: Becoming Leader. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:28.567183 12484 consensus_queue.cc:237] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.573700 12007 catalog_manager.cc:5654] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 reported cstate change: term changed from 3 to 4, leader changed from 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193) to 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194). New cstate: current_term: 4 leader_uuid: "3270daa0942c4603b01ef4ddab75ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:28.666453 12471 raft_consensus.cc:993] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6: : Instructing follower 3270daa0942c4603b01ef4ddab75ec12 to start an election
I20260202 15:07:28.666826 12481 raft_consensus.cc:1081] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Signalling peer 3270daa0942c4603b01ef4ddab75ec12 to start an election
I20260202 15:07:28.667884 12181 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd"
dest_uuid: "3270daa0942c4603b01ef4ddab75ec12"
from {username='slave'} at 127.0.0.1:40224
I20260202 15:07:28.668256 12181 raft_consensus.cc:493] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:07:28.668525 12181 raft_consensus.cc:3060] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:28.673122 12181 raft_consensus.cc:515] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.674376 12181 leader_election.cc:290] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 3 election: Requested vote from peers 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833), 0776af4ab68c4c79a6110f2f2a81dc61 (127.11.46.195:40889)
I20260202 15:07:28.674983 12107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "93e91e2eb8834e7b8d844526093e59d6"
I20260202 15:07:28.675155 12258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fa1440387614a6a9c30b73df4c211fd" candidate_uuid: "3270daa0942c4603b01ef4ddab75ec12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0776af4ab68c4c79a6110f2f2a81dc61"
I20260202 15:07:28.675458 12107 raft_consensus.cc:3055] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:07:28.675536 12258 raft_consensus.cc:3060] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:28.675768 12107 raft_consensus.cc:740] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93e91e2eb8834e7b8d844526093e59d6, State: Running, Role: LEADER
I20260202 15:07:28.676461 12107 consensus_queue.cc:260] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.677527 12107 raft_consensus.cc:3060] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:07:28.680924 12258 raft_consensus.cc:2468] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 3.
I20260202 15:07:28.681672 12150 leader_election.cc:304] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776af4ab68c4c79a6110f2f2a81dc61, 3270daa0942c4603b01ef4ddab75ec12; no voters:
I20260202 15:07:28.682199 12484 raft_consensus.cc:2804] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:07:28.682588 12484 raft_consensus.cc:697] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 3 LEADER]: Becoming Leader. State: Replica: 3270daa0942c4603b01ef4ddab75ec12, State: Running, Role: LEADER
I20260202 15:07:28.683070 12107 raft_consensus.cc:2468] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3270daa0942c4603b01ef4ddab75ec12 in term 3.
I20260202 15:07:28.683401 12484 consensus_queue.cc:237] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } }
I20260202 15:07:28.688640 12007 catalog_manager.cc:5654] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 reported cstate change: term changed from 2 to 3, leader changed from 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193) to 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194). New cstate: current_term: 3 leader_uuid: "3270daa0942c4603b01ef4ddab75ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:28.879086 12181 raft_consensus.cc:1275] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:28.880263 12479 consensus_queue.cc:1048] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:28.888034 12107 raft_consensus.cc:1275] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:28.889976 12485 consensus_queue.cc:1048] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:28.904224 12181 raft_consensus.cc:1275] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:07:28.906599 12497 consensus_queue.cc:1048] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3270daa0942c4603b01ef4ddab75ec12" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260202 15:07:28.915548 12107 raft_consensus.cc:1275] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Refusing update from remote peer 0776af4ab68c4c79a6110f2f2a81dc61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:07:28.916532 12485 consensus_queue.cc:1048] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:29.087986 12258 raft_consensus.cc:1275] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Refusing update from remote peer 3270daa0942c4603b01ef4ddab75ec12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:29.089075 12484 consensus_queue.cc:1048] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:29.096338 12107 raft_consensus.cc:1275] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Refusing update from remote peer 3270daa0942c4603b01ef4ddab75ec12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:07:29.097609 12492 consensus_queue.cc:1048] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:29.112244 12107 raft_consensus.cc:1275] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Refusing update from remote peer 3270daa0942c4603b01ef4ddab75ec12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:07:29.113003 12492 consensus_queue.cc:1048] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e91e2eb8834e7b8d844526093e59d6" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:29.120208 12258 raft_consensus.cc:1275] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Refusing update from remote peer 3270daa0942c4603b01ef4ddab75ec12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:07:29.121246 12484 consensus_queue.cc:1048] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0776af4ab68c4c79a6110f2f2a81dc61" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:29.281037 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:29.286424 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:29.286774 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:30.287565 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:30.292227 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:30.292574 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:31.293406 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:31.299204 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:31.299585 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:32.300418 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:32.305888 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:32.306277 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:33.307018 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:33.311619 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:33.312038 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:34.312922 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:34.317946 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:34.318279 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:35.319046 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:35.323599 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:35.323930 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:36.324693 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:36.329321 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:36.329694 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:37.330505 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:07:37.335281 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260202 15:07:37.335606 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:07:38.362933 11451 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20260202 15:07:38.363479 11451 tablet_server.cc:178] TabletServer@127.11.46.193:0 shutting down...
I20260202 15:07:38.390303 11451 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260202 15:07:38.391096 11451 tablet_replica.cc:333] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.392207 11451 raft_consensus.cc:2243] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.393009 11451 raft_consensus.cc:2272] T 58af84d12bfe4f359253736f8bbdce4b P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.395120 11451 tablet_replica.cc:333] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.395555 11451 raft_consensus.cc:2243] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.395852 11451 raft_consensus.cc:2272] T 9b2521523b414554957817815d71e97f P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.397279 11451 tablet_replica.cc:333] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.397655 11451 raft_consensus.cc:2243] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.398377 11451 raft_consensus.cc:2272] T 763ffb7935a149728615aec0b0f953f7 P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.399917 11451 tablet_replica.cc:333] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.400352 11451 raft_consensus.cc:2243] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.400770 11451 raft_consensus.cc:2272] T 05b935b29d684295a2d3587ff36f921e P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.402297 11451 tablet_replica.cc:333] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.402863 11451 raft_consensus.cc:2243] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 2 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.403745 11451 raft_consensus.cc:2272] T c0ebeba16c704c3eb18cacfbfc74d95e P 93e91e2eb8834e7b8d844526093e59d6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260202 15:07:38.405050 12149 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
I20260202 15:07:38.406531 11451 tablet_replica.cc:333] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.407402 11451 raft_consensus.cc:2243] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.407944 11451 raft_consensus.cc:2272] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260202 15:07:38.408896 12149 consensus_peers.cc:597] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 -> Peer 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833): Couldn't send request to peer 93e91e2eb8834e7b8d844526093e59d6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:38.409989 11451 tablet_replica.cc:333] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.410424 11451 raft_consensus.cc:2243] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.410854 11451 raft_consensus.cc:2272] T 0fa1440387614a6a9c30b73df4c211fd P 93e91e2eb8834e7b8d844526093e59d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.412261 11451 tablet_replica.cc:333] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.412643 11451 raft_consensus.cc:2243] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.412972 11451 raft_consensus.cc:2272] T 68bb2b68bcfb46aca45082c97b6b4ace P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.414791 11451 tablet_replica.cc:333] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.415331 11451 raft_consensus.cc:2243] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 4 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.416039 11451 raft_consensus.cc:2272] T 11f27edb4a944789ade0124dbf15d314 P 93e91e2eb8834e7b8d844526093e59d6 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260202 15:07:38.417516 12223 consensus_peers.cc:597] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 -> Peer 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833): Couldn't send request to peer 93e91e2eb8834e7b8d844526093e59d6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:38.417831 11451 tablet_replica.cc:333] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6: stopping tablet replica
I20260202 15:07:38.418244 11451 raft_consensus.cc:2243] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.418881 11451 raft_consensus.cc:2272] T d38bd74acf9241bc8aec9c60f7ab3924 P 93e91e2eb8834e7b8d844526093e59d6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260202 15:07:38.434610 12223 consensus_peers.cc:597] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 -> Peer 93e91e2eb8834e7b8d844526093e59d6 (127.11.46.193:40833): Couldn't send request to peer 93e91e2eb8834e7b8d844526093e59d6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:38.449174 11451 tablet_server.cc:195] TabletServer@127.11.46.193:0 shutdown complete.
I20260202 15:07:38.467096 11451 tablet_server.cc:178] TabletServer@127.11.46.194:0 shutting down...
I20260202 15:07:38.491048 11451 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260202 15:07:38.491739 11451 tablet_replica.cc:333] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
I20260202 15:07:38.492699 11451 raft_consensus.cc:2243] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.493315 11451 raft_consensus.cc:2272] T 05b935b29d684295a2d3587ff36f921e P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.495397 11451 tablet_replica.cc:333] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
I20260202 15:07:38.495877 11451 raft_consensus.cc:2243] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260202 15:07:38.496052 12221 consensus_peers.cc:597] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 -> Peer 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227): Couldn't send request to peer 3270daa0942c4603b01ef4ddab75ec12. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:38.496397 11451 raft_consensus.cc:2272] T 58af84d12bfe4f359253736f8bbdce4b P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.497964 11451 tablet_replica.cc:333] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
I20260202 15:07:38.498438 11451 raft_consensus.cc:2243] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.498852 11451 raft_consensus.cc:2272] T 11f27edb4a944789ade0124dbf15d314 P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.500464 11451 tablet_replica.cc:333] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
I20260202 15:07:38.500950 11451 raft_consensus.cc:2243] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.501313 11451 raft_consensus.cc:2272] T d38bd74acf9241bc8aec9c60f7ab3924 P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.502923 11451 tablet_replica.cc:333] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
I20260202 15:07:38.503463 11451 raft_consensus.cc:2243] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.503974 11451 raft_consensus.cc:2272] T 763ffb7935a149728615aec0b0f953f7 P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.505939 11451 tablet_replica.cc:333] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
W20260202 15:07:38.505878 12221 consensus_peers.cc:597] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 -> Peer 3270daa0942c4603b01ef4ddab75ec12 (127.11.46.194:43227): Couldn't send request to peer 3270daa0942c4603b01ef4ddab75ec12. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:38.506522 11451 raft_consensus.cc:2243] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 4 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.507229 11451 raft_consensus.cc:2272] T 68bb2b68bcfb46aca45082c97b6b4ace P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.508625 11451 tablet_replica.cc:333] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
I20260202 15:07:38.509057 11451 raft_consensus.cc:2243] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.509449 11451 raft_consensus.cc:2272] T c0ebeba16c704c3eb18cacfbfc74d95e P 3270daa0942c4603b01ef4ddab75ec12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.511065 11451 tablet_replica.cc:333] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
I20260202 15:07:38.511474 11451 raft_consensus.cc:2243] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 3 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.512162 11451 raft_consensus.cc:2272] T 0fa1440387614a6a9c30b73df4c211fd P 3270daa0942c4603b01ef4ddab75ec12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.513624 11451 tablet_replica.cc:333] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
I20260202 15:07:38.514142 11451 raft_consensus.cc:2243] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.514560 11451 raft_consensus.cc:2272] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 3270daa0942c4603b01ef4ddab75ec12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.515959 11451 tablet_replica.cc:333] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12: stopping tablet replica
I20260202 15:07:38.516314 11451 raft_consensus.cc:2243] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.516975 11451 raft_consensus.cc:2272] T 9b2521523b414554957817815d71e97f P 3270daa0942c4603b01ef4ddab75ec12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.543666 11451 tablet_server.cc:195] TabletServer@127.11.46.194:0 shutdown complete.
I20260202 15:07:38.561954 11451 tablet_server.cc:178] TabletServer@127.11.46.195:0 shutting down...
I20260202 15:07:38.583761 11451 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260202 15:07:38.584385 11451 tablet_replica.cc:333] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.584964 11451 raft_consensus.cc:2243] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.585698 11451 raft_consensus.cc:2272] T 05b935b29d684295a2d3587ff36f921e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.587292 11451 tablet_replica.cc:333] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.587710 11451 raft_consensus.cc:2243] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.588111 11451 raft_consensus.cc:2272] T 11f27edb4a944789ade0124dbf15d314 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.589480 11451 tablet_replica.cc:333] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.589965 11451 raft_consensus.cc:2243] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.590348 11451 raft_consensus.cc:2272] T d38bd74acf9241bc8aec9c60f7ab3924 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.591787 11451 tablet_replica.cc:333] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.592207 11451 raft_consensus.cc:2243] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.592571 11451 raft_consensus.cc:2272] T c0ebeba16c704c3eb18cacfbfc74d95e P 0776af4ab68c4c79a6110f2f2a81dc61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.593958 11451 tablet_replica.cc:333] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.594425 11451 raft_consensus.cc:2243] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.594892 11451 raft_consensus.cc:2272] T 763ffb7935a149728615aec0b0f953f7 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.596226 11451 tablet_replica.cc:333] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.596632 11451 raft_consensus.cc:2243] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.597002 11451 raft_consensus.cc:2272] T 68bb2b68bcfb46aca45082c97b6b4ace P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.598315 11451 tablet_replica.cc:333] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.598726 11451 raft_consensus.cc:2243] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.599431 11451 raft_consensus.cc:2272] T 58af84d12bfe4f359253736f8bbdce4b P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.601644 11451 tablet_replica.cc:333] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.602239 11451 raft_consensus.cc:2243] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.602598 11451 raft_consensus.cc:2272] T 9b2521523b414554957817815d71e97f P 0776af4ab68c4c79a6110f2f2a81dc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.604053 11451 tablet_replica.cc:333] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.604439 11451 raft_consensus.cc:2243] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.605104 11451 raft_consensus.cc:2272] T c74740e8b5ef4ff8b61d37b275dd4dc1 P 0776af4ab68c4c79a6110f2f2a81dc61 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.606418 11451 tablet_replica.cc:333] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61: stopping tablet replica
I20260202 15:07:38.606859 11451 raft_consensus.cc:2243] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:38.607280 11451 raft_consensus.cc:2272] T 0fa1440387614a6a9c30b73df4c211fd P 0776af4ab68c4c79a6110f2f2a81dc61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.649457 11451 tablet_server.cc:195] TabletServer@127.11.46.195:0 shutdown complete.
I20260202 15:07:38.665935 11451 master.cc:561] Master@127.11.46.254:37587 shutting down...
I20260202 15:07:38.688828 11451 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:07:38.689392 11451 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:38.689738 11451 tablet_replica.cc:333] T 00000000000000000000000000000000 P 52f9dbde7f6f418e96c88e824b0f3ba5: stopping tablet replica
I20260202 15:07:38.723078 11451 master.cc:583] Master@127.11.46.254:37587 shutdown complete.
0776af4ab68c4c79a6110f2f2a81dc61 1
3270daa0942c4603b01ef4ddab75ec12 4
93e91e2eb8834e7b8d844526093e59d6 5
0776af4ab68c4c79a6110f2f2a81dc61 3
3270daa0942c4603b01ef4ddab75ec12 4
93e91e2eb8834e7b8d844526093e59d6 3
0776af4ab68c4c79a6110f2f2a81dc61 1
3270daa0942c4603b01ef4ddab75ec12 1
93e91e2eb8834e7b8d844526093e59d6 8
0776af4ab68c4c79a6110f2f2a81dc61 3
3270daa0942c4603b01ef4ddab75ec12 3
93e91e2eb8834e7b8d844526093e59d6 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (30123 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260202 15:07:38.761577 11451 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.46.254:37385
I20260202 15:07:38.762457 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:07:38.767612 12523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:38.767681 12522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:38.768334 12525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:38.769654 11451 server_base.cc:1060] running on GCE node
I20260202 15:07:38.770642 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:07:38.770864 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:07:38.771025 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044858770999 us; error 0 us; skew 500 ppm
I20260202 15:07:38.771481 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:07:38.773543 11451 webserver.cc:492] Webserver started at http://127.11.46.254:39925/ using document root <none> and password file <none>
I20260202 15:07:38.773973 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:07:38.774142 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:07:38.774385 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:07:38.775454 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/master-0-root/instance:
uuid: "ec70d329e653431585e347713be6ee46"
format_stamp: "Formatted at 2026-02-02 15:07:38 on dist-test-slave-rflp"
I20260202 15:07:38.779382 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:07:38.782269 12530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:38.782951 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:07:38.783234 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/master-0-root
uuid: "ec70d329e653431585e347713be6ee46"
format_stamp: "Formatted at 2026-02-02 15:07:38 on dist-test-slave-rflp"
I20260202 15:07:38.783468 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:07:38.802094 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:07:38.803220 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:07:38.838840 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.254:37385
I20260202 15:07:38.839008 12582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.254:37385 every 8 connection(s)
I20260202 15:07:38.842321 12583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:38.851365 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46: Bootstrap starting.
I20260202 15:07:38.855195 12583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:38.858557 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46: No bootstrap required, opened a new log
I20260202 15:07:38.860468 12583 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec70d329e653431585e347713be6ee46" member_type: VOTER }
I20260202 15:07:38.860855 12583 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:38.861025 12583 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec70d329e653431585e347713be6ee46, State: Initialized, Role: FOLLOWER
I20260202 15:07:38.861457 12583 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec70d329e653431585e347713be6ee46" member_type: VOTER }
I20260202 15:07:38.861795 12583 raft_consensus.cc:399] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:07:38.861951 12583 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:07:38.862138 12583 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:38.865445 12583 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec70d329e653431585e347713be6ee46" member_type: VOTER }
I20260202 15:07:38.865828 12583 leader_election.cc:304] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec70d329e653431585e347713be6ee46; no voters:
I20260202 15:07:38.866739 12583 leader_election.cc:290] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:07:38.867115 12586 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:38.868322 12586 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [term 1 LEADER]: Becoming Leader. State: Replica: ec70d329e653431585e347713be6ee46, State: Running, Role: LEADER
I20260202 15:07:38.868836 12586 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec70d329e653431585e347713be6ee46" member_type: VOTER }
I20260202 15:07:38.869283 12583 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:07:38.873497 12588 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec70d329e653431585e347713be6ee46. Latest consensus state: current_term: 1 leader_uuid: "ec70d329e653431585e347713be6ee46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec70d329e653431585e347713be6ee46" member_type: VOTER } }
I20260202 15:07:38.873943 12588 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [sys.catalog]: This master's current role is: LEADER
I20260202 15:07:38.873287 12587 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec70d329e653431585e347713be6ee46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec70d329e653431585e347713be6ee46" member_type: VOTER } }
I20260202 15:07:38.876291 12587 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46 [sys.catalog]: This master's current role is: LEADER
I20260202 15:07:38.877121 12594 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:07:38.881582 12594 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:07:38.883502 11451 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260202 15:07:38.890254 12594 catalog_manager.cc:1357] Generated new cluster ID: d3e361b6f37448c285199c3f5ef4663b
I20260202 15:07:38.890544 12594 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:07:38.910153 12594 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:07:38.911270 12594 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:07:38.920886 12594 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46: Generated new TSK 0
I20260202 15:07:38.921344 12594 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:07:38.953111 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:07:38.958194 12604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:38.959059 12605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:38.960479 12607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:38.960986 11451 server_base.cc:1060] running on GCE node
I20260202 15:07:38.961784 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:07:38.961966 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:07:38.962108 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044858962080 us; error 0 us; skew 500 ppm
I20260202 15:07:38.962570 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:07:38.964617 11451 webserver.cc:492] Webserver started at http://127.11.46.193:41179/ using document root <none> and password file <none>
I20260202 15:07:38.965062 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:07:38.965234 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:07:38.965459 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:07:38.966363 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/instance:
uuid: "bbf140958c3b4f339c512a9dc4be2dab"
format_stamp: "Formatted at 2026-02-02 15:07:38 on dist-test-slave-rflp"
I20260202 15:07:38.970175 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:07:38.972980 12612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:38.973623 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:07:38.973899 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root
uuid: "bbf140958c3b4f339c512a9dc4be2dab"
format_stamp: "Formatted at 2026-02-02 15:07:38 on dist-test-slave-rflp"
I20260202 15:07:38.974133 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:07:38.990655 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:07:38.991880 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:07:38.993158 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:07:38.995146 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:07:38.995347 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:38.995581 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:07:38.995746 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:39.032168 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.193:40057
I20260202 15:07:39.032303 12674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.193:40057 every 8 connection(s)
I20260202 15:07:39.035990 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:07:39.042866 12679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:39.043059 12680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:39.046371 12682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:39.046640 11451 server_base.cc:1060] running on GCE node
I20260202 15:07:39.048197 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:07:39.048452 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:07:39.048672 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044859048653 us; error 0 us; skew 500 ppm
I20260202 15:07:39.049147 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:07:39.051548 11451 webserver.cc:492] Webserver started at http://127.11.46.194:34183/ using document root <none> and password file <none>
I20260202 15:07:39.051800 12675 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37385
I20260202 15:07:39.052043 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:07:39.052253 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:07:39.052111 12675 heartbeater.cc:461] Registering TS with master...
I20260202 15:07:39.052505 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:07:39.052987 12675 heartbeater.cc:507] Master 127.11.46.254:37385 requested a full tablet report, sending...
I20260202 15:07:39.053470 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/instance:
uuid: "6921687ed3814cccb2bd9c606875b9f0"
format_stamp: "Formatted at 2026-02-02 15:07:39 on dist-test-slave-rflp"
I20260202 15:07:39.054865 12547 ts_manager.cc:194] Registered new tserver with Master: bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:07:39.057097 12547 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45920
I20260202 15:07:39.058919 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260202 15:07:39.061882 12687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:39.062542 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:07:39.062826 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root
uuid: "6921687ed3814cccb2bd9c606875b9f0"
format_stamp: "Formatted at 2026-02-02 15:07:39 on dist-test-slave-rflp"
I20260202 15:07:39.063091 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:07:39.071786 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:07:39.072796 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:07:39.074043 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:07:39.076037 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:07:39.076241 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:39.076444 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:07:39.076591 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:39.113762 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.194:44367
I20260202 15:07:39.113855 12750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.194:44367 every 8 connection(s)
I20260202 15:07:39.118131 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:07:39.124708 12754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:39.126091 12755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:39.127163 11451 server_base.cc:1060] running on GCE node
W20260202 15:07:39.128245 12758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:39.129246 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:07:39.129503 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:07:39.129712 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044859129691 us; error 0 us; skew 500 ppm
I20260202 15:07:39.130345 12751 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37385
I20260202 15:07:39.130429 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:07:39.130775 12751 heartbeater.cc:461] Registering TS with master...
I20260202 15:07:39.131585 12751 heartbeater.cc:507] Master 127.11.46.254:37385 requested a full tablet report, sending...
I20260202 15:07:39.132833 11451 webserver.cc:492] Webserver started at http://127.11.46.195:36255/ using document root <none> and password file <none>
I20260202 15:07:39.133208 12547 ts_manager.cc:194] Registered new tserver with Master: 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:39.133301 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:07:39.133586 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:07:39.133831 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:07:39.134466 12547 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45928
I20260202 15:07:39.134977 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/instance:
uuid: "1eca64246cad41379cde4e2f9c3e25d2"
format_stamp: "Formatted at 2026-02-02 15:07:39 on dist-test-slave-rflp"
I20260202 15:07:39.139037 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260202 15:07:39.141920 12762 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:39.142547 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260202 15:07:39.142830 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root
uuid: "1eca64246cad41379cde4e2f9c3e25d2"
format_stamp: "Formatted at 2026-02-02 15:07:39 on dist-test-slave-rflp"
I20260202 15:07:39.143085 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:07:39.155937 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:07:39.156879 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:07:39.158103 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:07:39.160090 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:07:39.160312 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:39.160545 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:07:39.160738 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:39.200682 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.195:45829
I20260202 15:07:39.200830 12824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.195:45829 every 8 connection(s)
I20260202 15:07:39.217695 12825 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37385
I20260202 15:07:39.218065 12825 heartbeater.cc:461] Registering TS with master...
I20260202 15:07:39.218969 12825 heartbeater.cc:507] Master 127.11.46.254:37385 requested a full tablet report, sending...
I20260202 15:07:39.221487 12547 ts_manager.cc:194] Registered new tserver with Master: 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:39.222380 11451 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017408157s
I20260202 15:07:39.222930 12547 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45944
I20260202 15:07:39.248406 12547 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45952:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:07:39.254199 12547 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:07:39.447157 12716 tablet_service.cc:1505] Processing CreateTablet for tablet a3e391e37e424b1b8b2c63352ec8c198 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260202 15:07:39.448506 12716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e391e37e424b1b8b2c63352ec8c198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.451699 12712 tablet_service.cc:1505] Processing CreateTablet for tablet 65f098e59c0240a3a65cbd15dc358b64 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260202 15:07:39.452633 12640 tablet_service.cc:1505] Processing CreateTablet for tablet a3e391e37e424b1b8b2c63352ec8c198 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260202 15:07:39.453050 12712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f098e59c0240a3a65cbd15dc358b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.453840 12640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e391e37e424b1b8b2c63352ec8c198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.454349 12635 tablet_service.cc:1505] Processing CreateTablet for tablet ebc29e67ee294975abcdfc9e53f8b6e1 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260202 15:07:39.454516 12790 tablet_service.cc:1505] Processing CreateTablet for tablet a3e391e37e424b1b8b2c63352ec8c198 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260202 15:07:39.455458 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc29e67ee294975abcdfc9e53f8b6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.455751 12790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e391e37e424b1b8b2c63352ec8c198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.455803 12711 tablet_service.cc:1505] Processing CreateTablet for tablet 4ed5f5e04d064d6e8d431aa2209cc141 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260202 15:07:39.457125 12711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed5f5e04d064d6e8d431aa2209cc141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.456816 12638 tablet_service.cc:1505] Processing CreateTablet for tablet e4d8302d577042feaa44f63d596dc87d (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260202 15:07:39.458112 12638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d8302d577042feaa44f63d596dc87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.459899 12637 tablet_service.cc:1505] Processing CreateTablet for tablet 65f098e59c0240a3a65cbd15dc358b64 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260202 15:07:39.459964 12789 tablet_service.cc:1505] Processing CreateTablet for tablet dbc759e31af04c60b10baa9dce36e825 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260202 15:07:39.460353 12714 tablet_service.cc:1505] Processing CreateTablet for tablet e4d8302d577042feaa44f63d596dc87d (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260202 15:07:39.461189 12637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f098e59c0240a3a65cbd15dc358b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.461388 12789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbc759e31af04c60b10baa9dce36e825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.461568 12714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d8302d577042feaa44f63d596dc87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.463821 12636 tablet_service.cc:1505] Processing CreateTablet for tablet 4ed5f5e04d064d6e8d431aa2209cc141 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260202 15:07:39.463861 12788 tablet_service.cc:1505] Processing CreateTablet for tablet e4d8302d577042feaa44f63d596dc87d (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260202 15:07:39.463873 12715 tablet_service.cc:1505] Processing CreateTablet for tablet dbc759e31af04c60b10baa9dce36e825 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260202 15:07:39.465032 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed5f5e04d064d6e8d431aa2209cc141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.465191 12715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbc759e31af04c60b10baa9dce36e825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.466292 12787 tablet_service.cc:1505] Processing CreateTablet for tablet 65f098e59c0240a3a65cbd15dc358b64 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260202 15:07:39.468412 12786 tablet_service.cc:1505] Processing CreateTablet for tablet 4ed5f5e04d064d6e8d431aa2209cc141 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260202 15:07:39.455749 12639 tablet_service.cc:1505] Processing CreateTablet for tablet dbc759e31af04c60b10baa9dce36e825 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260202 15:07:39.471045 12785 tablet_service.cc:1505] Processing CreateTablet for tablet ebc29e67ee294975abcdfc9e53f8b6e1 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260202 15:07:39.476217 12710 tablet_service.cc:1505] Processing CreateTablet for tablet ebc29e67ee294975abcdfc9e53f8b6e1 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260202 15:07:39.477411 12710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc29e67ee294975abcdfc9e53f8b6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.472324 12639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbc759e31af04c60b10baa9dce36e825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.483934 12784 tablet_service.cc:1505] Processing CreateTablet for tablet 6ca9793f4e4e46418e44fbdc3d3a4794 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260202 15:07:39.485950 12783 tablet_service.cc:1505] Processing CreateTablet for tablet f8f84948a93c4dc984e69bfd1f44be23 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260202 15:07:39.487972 12782 tablet_service.cc:1505] Processing CreateTablet for tablet 9335cdc844e44d608b9cdd8369f9afb1 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260202 15:07:39.488505 12634 tablet_service.cc:1505] Processing CreateTablet for tablet 6ca9793f4e4e46418e44fbdc3d3a4794 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260202 15:07:39.489807 12634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca9793f4e4e46418e44fbdc3d3a4794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.489907 12781 tablet_service.cc:1505] Processing CreateTablet for tablet 1c6861781ee34db9b79870494af1cced (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260202 15:07:39.465191 12788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d8302d577042feaa44f63d596dc87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.491521 12784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca9793f4e4e46418e44fbdc3d3a4794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.492681 12709 tablet_service.cc:1505] Processing CreateTablet for tablet 6ca9793f4e4e46418e44fbdc3d3a4794 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260202 15:07:39.493891 12709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca9793f4e4e46418e44fbdc3d3a4794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.494998 12785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc29e67ee294975abcdfc9e53f8b6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.496279 12708 tablet_service.cc:1505] Processing CreateTablet for tablet f8f84948a93c4dc984e69bfd1f44be23 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260202 15:07:39.497380 12708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f84948a93c4dc984e69bfd1f44be23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.498394 12787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f098e59c0240a3a65cbd15dc358b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.499655 12707 tablet_service.cc:1505] Processing CreateTablet for tablet 9335cdc844e44d608b9cdd8369f9afb1 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260202 15:07:39.500850 12707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9335cdc844e44d608b9cdd8369f9afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.503743 12706 tablet_service.cc:1505] Processing CreateTablet for tablet 1c6861781ee34db9b79870494af1cced (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260202 15:07:39.505432 12706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6861781ee34db9b79870494af1cced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.508050 12782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9335cdc844e44d608b9cdd8369f9afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.511003 12783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f84948a93c4dc984e69bfd1f44be23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.513298 12781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6861781ee34db9b79870494af1cced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.516160 12786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed5f5e04d064d6e8d431aa2209cc141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.544624 12633 tablet_service.cc:1505] Processing CreateTablet for tablet f8f84948a93c4dc984e69bfd1f44be23 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260202 15:07:39.545893 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f84948a93c4dc984e69bfd1f44be23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.558018 12710 tablet_service.cc:1505] Processing CreateTablet for tablet e9b34ca86c2f4e9b831f13ebf9cd3aa8 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260202 15:07:39.567716 12632 tablet_service.cc:1505] Processing CreateTablet for tablet 1c6861781ee34db9b79870494af1cced (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260202 15:07:39.568769 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6861781ee34db9b79870494af1cced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.571852 12631 tablet_service.cc:1505] Processing CreateTablet for tablet e9b34ca86c2f4e9b831f13ebf9cd3aa8 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260202 15:07:39.572942 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b34ca86c2f4e9b831f13ebf9cd3aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.575726 12640 tablet_service.cc:1505] Processing CreateTablet for tablet 9335cdc844e44d608b9cdd8369f9afb1 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260202 15:07:39.576865 12640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9335cdc844e44d608b9cdd8369f9afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.576699 12711 tablet_service.cc:1505] Processing CreateTablet for tablet 9592591cd602455aa52fdc1d84e48b63 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260202 15:07:39.577788 12711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9592591cd602455aa52fdc1d84e48b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.587301 12785 tablet_service.cc:1505] Processing CreateTablet for tablet 3a02578a734446c387f55d747a3f3bd0 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260202 15:07:39.588375 12785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a02578a734446c387f55d747a3f3bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.592760 12784 tablet_service.cc:1505] Processing CreateTablet for tablet e9b34ca86c2f4e9b831f13ebf9cd3aa8 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260202 15:07:39.593894 12784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b34ca86c2f4e9b831f13ebf9cd3aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.594098 12786 tablet_service.cc:1505] Processing CreateTablet for tablet 9592591cd602455aa52fdc1d84e48b63 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260202 15:07:39.605767 12842 tablet_bootstrap.cc:492] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:39.607604 12716 tablet_service.cc:1505] Processing CreateTablet for tablet 4d68e0d1324241f1878c450b36fac925 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260202 15:07:39.608799 12716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d68e0d1324241f1878c450b36fac925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.611898 12842 tablet_bootstrap.cc:654] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:39.612174 12708 tablet_service.cc:1505] Processing CreateTablet for tablet cb4cc1899a25452c9c57398fe4022645 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260202 15:07:39.612628 12710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b34ca86c2f4e9b831f13ebf9cd3aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.613375 12708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4cc1899a25452c9c57398fe4022645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.615073 12714 tablet_service.cc:1505] Processing CreateTablet for tablet 3b57a31e08c1420ab954d13d66af9297 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260202 15:07:39.615680 12712 tablet_service.cc:1505] Processing CreateTablet for tablet 78ef01280730461580740297403fe090 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260202 15:07:39.616515 12714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b57a31e08c1420ab954d13d66af9297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.617275 12841 tablet_bootstrap.cc:492] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:39.632229 12636 tablet_service.cc:1505] Processing CreateTablet for tablet cb4cc1899a25452c9c57398fe4022645 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260202 15:07:39.633919 12781 tablet_service.cc:1505] Processing CreateTablet for tablet cb4cc1899a25452c9c57398fe4022645 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260202 15:07:39.638010 12781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4cc1899a25452c9c57398fe4022645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.637861 12638 tablet_service.cc:1505] Processing CreateTablet for tablet 3b57a31e08c1420ab954d13d66af9297 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260202 15:07:39.636397 12790 tablet_service.cc:1505] Processing CreateTablet for tablet 3b57a31e08c1420ab954d13d66af9297 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260202 15:07:39.641840 12790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b57a31e08c1420ab954d13d66af9297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.636256 12706 tablet_service.cc:1505] Processing CreateTablet for tablet f72d171c95a149d890432428ecf80739 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260202 15:07:39.642932 12789 tablet_service.cc:1505] Processing CreateTablet for tablet 78ef01280730461580740297403fe090 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260202 15:07:39.648020 12789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ef01280730461580740297403fe090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.647707 12787 tablet_service.cc:1505] Processing CreateTablet for tablet f72d171c95a149d890432428ecf80739 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260202 15:07:39.654729 12783 tablet_service.cc:1505] Processing CreateTablet for tablet 4d68e0d1324241f1878c450b36fac925 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260202 15:07:39.655413 12787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f72d171c95a149d890432428ecf80739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.659179 12786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9592591cd602455aa52fdc1d84e48b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.667974 12782 tablet_service.cc:1505] Processing CreateTablet for tablet 015f643434d84eac8a3774b663eb4220 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260202 15:07:39.669164 12782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015f643434d84eac8a3774b663eb4220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.620350 12637 tablet_service.cc:1505] Processing CreateTablet for tablet 4d68e0d1324241f1878c450b36fac925 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260202 15:07:39.673925 12639 tablet_service.cc:1505] Processing CreateTablet for tablet f72d171c95a149d890432428ecf80739 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260202 15:07:39.674360 12783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d68e0d1324241f1878c450b36fac925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.675679 12635 tablet_service.cc:1505] Processing CreateTablet for tablet 3a02578a734446c387f55d747a3f3bd0 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260202 15:07:39.650050 12633 tablet_service.cc:1505] Processing CreateTablet for tablet 78ef01280730461580740297403fe090 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260202 15:07:39.681082 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4cc1899a25452c9c57398fe4022645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.685070 12715 tablet_service.cc:1505] Processing CreateTablet for tablet 3a02578a734446c387f55d747a3f3bd0 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260202 15:07:39.704203 12842 tablet_bootstrap.cc:492] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:39.704671 12842 ts_tablet_manager.cc:1403] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.099s user 0.047s sys 0.028s
I20260202 15:07:39.686527 12640 tablet_service.cc:1505] Processing CreateTablet for tablet 015f643434d84eac8a3774b663eb4220 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260202 15:07:39.706804 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ef01280730461580740297403fe090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.707855 12709 tablet_service.cc:1505] Processing CreateTablet for tablet 015f643434d84eac8a3774b663eb4220 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260202 15:07:39.710425 12840 tablet_bootstrap.cc:492] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:39.710314 12707 tablet_service.cc:1505] Processing CreateTablet for tablet 9fae829b1c574bc795c5a424036329b3 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260202 15:07:39.711072 12634 tablet_service.cc:1505] Processing CreateTablet for tablet 9592591cd602455aa52fdc1d84e48b63 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260202 15:07:39.713195 12634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9592591cd602455aa52fdc1d84e48b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.713892 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a02578a734446c387f55d747a3f3bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.689801 12712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ef01280730461580740297403fe090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.715224 12706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f72d171c95a149d890432428ecf80739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.716444 12715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a02578a734446c387f55d747a3f3bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.707119 12842 raft_consensus.cc:359] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:39.718202 12637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d68e0d1324241f1878c450b36fac925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.720502 12789 tablet_service.cc:1505] Processing CreateTablet for tablet 4687c40305f64dd2960fcbb336f93395 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260202 15:07:39.721580 12789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4687c40305f64dd2960fcbb336f93395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.723006 12639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f72d171c95a149d890432428ecf80739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.727082 12638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b57a31e08c1420ab954d13d66af9297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.678558 12788 tablet_service.cc:1505] Processing CreateTablet for tablet 9fae829b1c574bc795c5a424036329b3 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260202 15:07:39.735363 12788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fae829b1c574bc795c5a424036329b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.736045 12711 tablet_service.cc:1505] Processing CreateTablet for tablet d30f8f27ce00483c85fb1758a4f81f03 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260202 15:07:39.737877 12782 tablet_service.cc:1505] Processing CreateTablet for tablet 9e78bd8179024ae0aca19878b91342a9 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260202 15:07:39.740209 12640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015f643434d84eac8a3774b663eb4220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.715684 12790 tablet_service.cc:1505] Processing CreateTablet for tablet d30f8f27ce00483c85fb1758a4f81f03 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260202 15:07:39.747119 12707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fae829b1c574bc795c5a424036329b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.748149 12716 tablet_service.cc:1505] Processing CreateTablet for tablet 4687c40305f64dd2960fcbb336f93395 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260202 15:07:39.750970 12709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015f643434d84eac8a3774b663eb4220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.752565 12841 tablet_bootstrap.cc:654] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:39.717749 12842 raft_consensus.cc:385] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:39.754913 12842 raft_consensus.cc:740] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:39.755721 12842 consensus_queue.cc:260] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:39.760545 12840 tablet_bootstrap.cc:654] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:39.761893 12782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e78bd8179024ae0aca19878b91342a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.763867 12784 tablet_service.cc:1505] Processing CreateTablet for tablet dc5a4c8371434c48926f011a134d3bd4 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260202 15:07:39.763012 12710 tablet_service.cc:1505] Processing CreateTablet for tablet dc5a4c8371434c48926f011a134d3bd4 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260202 15:07:39.763783 12716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4687c40305f64dd2960fcbb336f93395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.767192 12785 tablet_service.cc:1505] Processing CreateTablet for tablet d3a3797f335d43dca87344d8954a1158 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260202 15:07:39.771548 12787 tablet_service.cc:1505] Processing CreateTablet for tablet 69e93e86b75b464fa4c55fdeb141acea (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260202 15:07:39.771679 12714 tablet_service.cc:1505] Processing CreateTablet for tablet 9e78bd8179024ae0aca19878b91342a9 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260202 15:07:39.773622 12784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc5a4c8371434c48926f011a134d3bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.776090 12710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc5a4c8371434c48926f011a134d3bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.783963 12781 tablet_service.cc:1505] Processing CreateTablet for tablet 726983b999ef44c5a29a3f5d7461d92f (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260202 15:07:39.785174 12781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726983b999ef44c5a29a3f5d7461d92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.785808 12714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e78bd8179024ae0aca19878b91342a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.786649 12785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a3797f335d43dca87344d8954a1158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.787317 12711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30f8f27ce00483c85fb1758a4f81f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.787542 12790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30f8f27ce00483c85fb1758a4f81f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.791253 12787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e93e86b75b464fa4c55fdeb141acea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.791999 12634 tablet_service.cc:1505] Processing CreateTablet for tablet 9fae829b1c574bc795c5a424036329b3 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260202 15:07:39.793359 12634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fae829b1c574bc795c5a424036329b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.796758 12633 tablet_service.cc:1505] Processing CreateTablet for tablet d30f8f27ce00483c85fb1758a4f81f03 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260202 15:07:39.797015 12637 tablet_service.cc:1505] Processing CreateTablet for tablet d3a3797f335d43dca87344d8954a1158 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260202 15:07:39.797997 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30f8f27ce00483c85fb1758a4f81f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.798646 12631 tablet_service.cc:1505] Processing CreateTablet for tablet 4687c40305f64dd2960fcbb336f93395 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260202 15:07:39.800664 12825 heartbeater.cc:499] Master 127.11.46.254:37385 was elected leader, sending a full tablet report...
I20260202 15:07:39.813849 12783 tablet_service.cc:1505] Processing CreateTablet for tablet a739ed9c340d4dd9935861456bc3f3d3 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260202 15:07:39.800664 12847 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:39.830839 12847 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:39.832988 12847 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:39.832742 12632 tablet_service.cc:1505] Processing CreateTablet for tablet dc5a4c8371434c48926f011a134d3bd4 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260202 15:07:39.847054 12637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a3797f335d43dca87344d8954a1158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.851111 12788 tablet_service.cc:1505] Processing CreateTablet for tablet 5a776a09df954aa6ac5c756e8da6c56a (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260202 15:07:39.858696 12783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a739ed9c340d4dd9935861456bc3f3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.859661 12716 tablet_service.cc:1505] Processing CreateTablet for tablet d3a3797f335d43dca87344d8954a1158 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260202 15:07:39.863659 12788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a776a09df954aa6ac5c756e8da6c56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.864694 12716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a3797f335d43dca87344d8954a1158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.868232 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc5a4c8371434c48926f011a134d3bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.884070 12841 tablet_bootstrap.cc:492] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:39.884487 12841 ts_tablet_manager.cc:1403] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.268s user 0.002s sys 0.016s
W20260202 15:07:39.885492 12596 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:07:39.888231 12639 tablet_service.cc:1505] Processing CreateTablet for tablet 69e93e86b75b464fa4c55fdeb141acea (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260202 15:07:39.889443 12639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e93e86b75b464fa4c55fdeb141acea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.875941 12707 tablet_service.cc:1505] Processing CreateTablet for tablet 69e93e86b75b464fa4c55fdeb141acea (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260202 15:07:39.895583 12707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e93e86b75b464fa4c55fdeb141acea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.895987 12712 tablet_service.cc:1505] Processing CreateTablet for tablet 726983b999ef44c5a29a3f5d7461d92f (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260202 15:07:39.897125 12712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726983b999ef44c5a29a3f5d7461d92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.897923 12708 tablet_service.cc:1505] Processing CreateTablet for tablet a739ed9c340d4dd9935861456bc3f3d3 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260202 15:07:39.899029 12841 raft_consensus.cc:359] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:39.899708 12841 raft_consensus.cc:385] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:39.900009 12841 raft_consensus.cc:740] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:39.900691 12841 consensus_queue.cc:260] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:39.899515 12708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a739ed9c340d4dd9935861456bc3f3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.901453 12635 tablet_service.cc:1505] Processing CreateTablet for tablet 9e78bd8179024ae0aca19878b91342a9 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260202 15:07:39.917032 12842 ts_tablet_manager.cc:1434] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.212s user 0.006s sys 0.010s
I20260202 15:07:39.914063 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4687c40305f64dd2960fcbb336f93395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.934782 12640 tablet_service.cc:1505] Processing CreateTablet for tablet 5a776a09df954aa6ac5c756e8da6c56a (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260202 15:07:39.954026 12640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a776a09df954aa6ac5c756e8da6c56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.942457 12786 tablet_service.cc:1505] Processing CreateTablet for tablet ee143496d46d4a45a2d24f0990e74f44 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260202 15:07:39.916021 12634 tablet_service.cc:1505] Processing CreateTablet for tablet a739ed9c340d4dd9935861456bc3f3d3 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260202 15:07:39.956050 12634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a739ed9c340d4dd9935861456bc3f3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.957566 12782 tablet_service.cc:1505] Processing CreateTablet for tablet 85ea3f0239bb4603a3a363108154ef77 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260202 15:07:39.958806 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e78bd8179024ae0aca19878b91342a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.950358 12638 tablet_service.cc:1505] Processing CreateTablet for tablet 0d12e74920dc474599bb20f8e977a6b4 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260202 15:07:39.936252 12785 tablet_service.cc:1505] Processing CreateTablet for tablet 0d12e74920dc474599bb20f8e977a6b4 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260202 15:07:39.943971 12706 tablet_service.cc:1505] Processing CreateTablet for tablet 00b1a15d2d1246e1abe52586ddf512ef (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
W20260202 15:07:39.959403 12826 tablet.cc:2378] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:07:39.960279 12706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b1a15d2d1246e1abe52586ddf512ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.962960 12786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee143496d46d4a45a2d24f0990e74f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.966035 12782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ea3f0239bb4603a3a363108154ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.969156 12785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d12e74920dc474599bb20f8e977a6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.912554 12636 tablet_service.cc:1505] Processing CreateTablet for tablet 726983b999ef44c5a29a3f5d7461d92f (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260202 15:07:39.973461 12714 tablet_service.cc:1505] Processing CreateTablet for tablet ee143496d46d4a45a2d24f0990e74f44 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260202 15:07:39.974526 12714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee143496d46d4a45a2d24f0990e74f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.974337 12784 tablet_service.cc:1505] Processing CreateTablet for tablet 00b1a15d2d1246e1abe52586ddf512ef (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260202 15:07:39.976428 12842 tablet_bootstrap.cc:492] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:39.912508 12709 tablet_service.cc:1505] Processing CreateTablet for tablet 0d12e74920dc474599bb20f8e977a6b4 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260202 15:07:39.901597 12710 tablet_service.cc:1505] Processing CreateTablet for tablet 5a776a09df954aa6ac5c756e8da6c56a (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260202 15:07:39.975456 12784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b1a15d2d1246e1abe52586ddf512ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.960415 12638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d12e74920dc474599bb20f8e977a6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:39.985898 12840 tablet_bootstrap.cc:492] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:39.986286 12840 ts_tablet_manager.cc:1403] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.276s user 0.018s sys 0.016s
I20260202 15:07:39.989078 12633 tablet_service.cc:1505] Processing CreateTablet for tablet 00b1a15d2d1246e1abe52586ddf512ef (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260202 15:07:39.991964 12840 raft_consensus.cc:359] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:39.995399 12840 raft_consensus.cc:385] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:39.995661 12840 raft_consensus.cc:740] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:39.996330 12840 consensus_queue.cc:260] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:39.997311 12632 tablet_service.cc:1505] Processing CreateTablet for tablet ee143496d46d4a45a2d24f0990e74f44 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260202 15:07:40.011483 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:07:40.012862 12765 leader_election.cc:343] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.014154 12637 tablet_service.cc:1505] Processing CreateTablet for tablet 85ea3f0239bb4603a3a363108154ef77 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260202 15:07:40.015298 12637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ea3f0239bb4603a3a363108154ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.024132 12715 tablet_service.cc:1505] Processing CreateTablet for tablet 85ea3f0239bb4603a3a363108154ef77 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260202 15:07:40.030242 12716 tablet_service.cc:1505] Processing CreateTablet for tablet 83fc6cc808d9450788aeecc86caa94bd (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260202 15:07:40.027220 12709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d12e74920dc474599bb20f8e977a6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.036464 12710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a776a09df954aa6ac5c756e8da6c56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.037458 12842 tablet_bootstrap.cc:654] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.038957 12715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ea3f0239bb4603a3a363108154ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.041343 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726983b999ef44c5a29a3f5d7461d92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.048144 12707 tablet_service.cc:1505] Processing CreateTablet for tablet a50a506e17f747e0a940bc3c982336c6 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260202 15:07:40.055045 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b1a15d2d1246e1abe52586ddf512ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.057271 12783 tablet_service.cc:1505] Processing CreateTablet for tablet a50a506e17f747e0a940bc3c982336c6 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260202 15:07:40.058290 12783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50a506e17f747e0a940bc3c982336c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.059090 12675 heartbeater.cc:499] Master 127.11.46.254:37385 was elected leader, sending a full tablet report...
I20260202 15:07:40.059166 12851 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:40.059831 12851 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.062260 12851 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:40.062165 12711 tablet_service.cc:1505] Processing CreateTablet for tablet d88e30bfa6e641749640dde238a2a8f8 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260202 15:07:40.065212 12711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88e30bfa6e641749640dde238a2a8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.080497 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee143496d46d4a45a2d24f0990e74f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.080700 12716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83fc6cc808d9450788aeecc86caa94bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.097127 12788 tablet_service.cc:1505] Processing CreateTablet for tablet d88e30bfa6e641749640dde238a2a8f8 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260202 15:07:40.098316 12788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88e30bfa6e641749640dde238a2a8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.098814 12841 ts_tablet_manager.cc:1434] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.214s user 0.024s sys 0.044s
I20260202 15:07:40.099725 12841 tablet_bootstrap.cc:492] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:40.100128 12751 heartbeater.cc:499] Master 127.11.46.254:37385 was elected leader, sending a full tablet report...
I20260202 15:07:40.105181 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:40.109081 12707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50a506e17f747e0a940bc3c982336c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.054502 12634 tablet_service.cc:1505] Processing CreateTablet for tablet 83fc6cc808d9450788aeecc86caa94bd (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
W20260202 15:07:40.114187 12763 leader_election.cc:343] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.114606 12763 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:40.115837 12844 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:40.117512 12781 tablet_service.cc:1505] Processing CreateTablet for tablet 83fc6cc808d9450788aeecc86caa94bd (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260202 15:07:40.118744 12781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83fc6cc808d9450788aeecc86caa94bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260202 15:07:40.155098 12531 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 75456.8us
I20260202 15:07:40.162187 12638 tablet_service.cc:1505] Processing CreateTablet for tablet a50a506e17f747e0a940bc3c982336c6 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260202 15:07:40.114359 12634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83fc6cc808d9450788aeecc86caa94bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.170680 12633 tablet_service.cc:1505] Processing CreateTablet for tablet d88e30bfa6e641749640dde238a2a8f8 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260202 15:07:40.179527 12708 tablet_service.cc:1505] Processing CreateTablet for tablet 0f70fb0b2ffc4a51b06fe26a22355100 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260202 15:07:40.177285 12842 tablet_bootstrap.cc:492] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:40.183921 12842 ts_tablet_manager.cc:1403] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.208s user 0.019s sys 0.009s
I20260202 15:07:40.184007 12708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f70fb0b2ffc4a51b06fe26a22355100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.186508 12842 raft_consensus.cc:359] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.187283 12842 raft_consensus.cc:385] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.187628 12842 raft_consensus.cc:740] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.188546 12842 consensus_queue.cc:260] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.187753 12840 ts_tablet_manager.cc:1434] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.201s user 0.021s sys 0.013s
I20260202 15:07:40.190734 12842 ts_tablet_manager.cc:1434] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:07:40.191447 12840 tablet_bootstrap.cc:492] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:40.192255 12842 tablet_bootstrap.cc:492] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:40.180531 12637 tablet_service.cc:1505] Processing CreateTablet for tablet 435e192b63f348a58c00c3ae9e598ea9 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260202 15:07:40.173652 12712 tablet_service.cc:1505] Processing CreateTablet for tablet 435e192b63f348a58c00c3ae9e598ea9 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260202 15:07:40.202145 12712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435e192b63f348a58c00c3ae9e598ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.202988 12638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50a506e17f747e0a940bc3c982336c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.207013 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88e30bfa6e641749640dde238a2a8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.182677 12639 tablet_service.cc:1505] Processing CreateTablet for tablet 0f70fb0b2ffc4a51b06fe26a22355100 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260202 15:07:40.197891 12841 tablet_bootstrap.cc:654] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.200840 12635 tablet_service.cc:1505] Processing CreateTablet for tablet 7d28e4999b504df4a53610187731416d (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260202 15:07:40.210140 12637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435e192b63f348a58c00c3ae9e598ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.222894 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d28e4999b504df4a53610187731416d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.226871 12639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f70fb0b2ffc4a51b06fe26a22355100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.265532 12711 tablet_service.cc:1505] Processing CreateTablet for tablet 7d28e4999b504df4a53610187731416d (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260202 15:07:40.266884 12711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d28e4999b504df4a53610187731416d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.266855 12708 tablet_service.cc:1505] Processing CreateTablet for tablet 4d879eebd1924ad8b6f65647f4e73362 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260202 15:07:40.268180 12708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d879eebd1924ad8b6f65647f4e73362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.271708 12716 tablet_service.cc:1505] Processing CreateTablet for tablet 01f20099bc6844498a65c943dde6b1ff (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260202 15:07:40.272806 12716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f20099bc6844498a65c943dde6b1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.279892 12706 tablet_service.cc:1505] Processing CreateTablet for tablet 323d926990494ad1aafb6aa0282d9f68 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260202 15:07:40.281162 12706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323d926990494ad1aafb6aa0282d9f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.288399 12715 tablet_service.cc:1505] Processing CreateTablet for tablet 30df58c1134a49e18c29202abd31da41 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260202 15:07:40.289481 12715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30df58c1134a49e18c29202abd31da41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.293247 12634 tablet_service.cc:1505] Processing CreateTablet for tablet 4d879eebd1924ad8b6f65647f4e73362 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260202 15:07:40.294283 12634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d879eebd1924ad8b6f65647f4e73362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.294369 12632 tablet_service.cc:1505] Processing CreateTablet for tablet 323d926990494ad1aafb6aa0282d9f68 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260202 15:07:40.295403 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323d926990494ad1aafb6aa0282d9f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.295935 12840 tablet_bootstrap.cc:654] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.302331 12631 tablet_service.cc:1505] Processing CreateTablet for tablet 6f9321e2251943229a0b7dbdc0577c6b (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260202 15:07:40.302536 12640 tablet_service.cc:1505] Processing CreateTablet for tablet 01f20099bc6844498a65c943dde6b1ff (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260202 15:07:40.303400 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9321e2251943229a0b7dbdc0577c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.307478 12636 tablet_service.cc:1505] Processing CreateTablet for tablet 30df58c1134a49e18c29202abd31da41 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260202 15:07:40.308456 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30df58c1134a49e18c29202abd31da41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.309036 12640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f20099bc6844498a65c943dde6b1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.312672 12711 tablet_service.cc:1505] Processing CreateTablet for tablet 6f9321e2251943229a0b7dbdc0577c6b (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260202 15:07:40.313687 12711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9321e2251943229a0b7dbdc0577c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.314810 12844 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:40.315301 12844 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.317440 12844 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:40.319329 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:40.319657 12710 tablet_service.cc:1505] Processing CreateTablet for tablet c4798df17a1e4d18a594bb1b37b98bae (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
W20260202 15:07:40.320818 12765 leader_election.cc:343] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.322146 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:40.322813 12650 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:40.324023 12709 tablet_service.cc:1505] Processing CreateTablet for tablet 87df3a9146bb41e1ac7468e17d2d9498 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260202 15:07:40.324126 12763 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:40.325285 12844 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:40.325641 12844 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:40.325966 12844 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:40.315316 12842 tablet_bootstrap.cc:654] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.350313 12852 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:40.350767 12852 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.352902 12852 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:40.320829 12710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4798df17a1e4d18a594bb1b37b98bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.365610 12709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87df3a9146bb41e1ac7468e17d2d9498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.340251 12844 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.395021 12633 tablet_service.cc:1505] Processing CreateTablet for tablet 6fc895893bbe4569bb328cfc3513de38 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260202 15:07:40.380556 12638 tablet_service.cc:1505] Processing CreateTablet for tablet c4798df17a1e4d18a594bb1b37b98bae (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260202 15:07:40.402846 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:40.405022 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc895893bbe4569bb328cfc3513de38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.408303 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:40.408783 12650 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:40.405257 12840 tablet_bootstrap.cc:492] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:40.411144 12844 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:07:40.411876 12615 leader_election.cc:343] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.412012 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:40.412897 12638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4798df17a1e4d18a594bb1b37b98bae. 1 dirs total, 0 dirs full, 0 dirs failed
W20260202 15:07:40.413093 12765 leader_election.cc:343] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.411028 12840 ts_tablet_manager.cc:1403] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.220s user 0.019s sys 0.014s
I20260202 15:07:40.416265 12841 tablet_bootstrap.cc:492] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:40.416733 12841 ts_tablet_manager.cc:1403] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.317s user 0.016s sys 0.014s
I20260202 15:07:40.420508 12650 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:40.402977 12708 tablet_service.cc:1505] Processing CreateTablet for tablet 6fc895893bbe4569bb328cfc3513de38 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260202 15:07:40.374397 12707 tablet_service.cc:1505] Processing CreateTablet for tablet d2476071e1d24aa6a1342c8d81699cef (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260202 15:07:40.416615 12635 tablet_service.cc:1505] Processing CreateTablet for tablet 87df3a9146bb41e1ac7468e17d2d9498 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260202 15:07:40.424010 12639 tablet_service.cc:1505] Processing CreateTablet for tablet d2476071e1d24aa6a1342c8d81699cef (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260202 15:07:40.425108 12840 raft_consensus.cc:359] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.425706 12840 raft_consensus.cc:385] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.425941 12840 raft_consensus.cc:740] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.429967 12782 tablet_service.cc:1505] Processing CreateTablet for tablet 4d879eebd1924ad8b6f65647f4e73362 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260202 15:07:40.428637 12781 tablet_service.cc:1505] Processing CreateTablet for tablet 0f70fb0b2ffc4a51b06fe26a22355100 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260202 15:07:40.448768 12782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d879eebd1924ad8b6f65647f4e73362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.449342 12781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f70fb0b2ffc4a51b06fe26a22355100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.449198 12790 tablet_service.cc:1505] Processing CreateTablet for tablet 30df58c1134a49e18c29202abd31da41 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260202 15:07:40.433861 12763 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:40.427760 12786 tablet_service.cc:1505] Processing CreateTablet for tablet 435e192b63f348a58c00c3ae9e598ea9 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260202 15:07:40.450446 12790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30df58c1134a49e18c29202abd31da41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.450742 12844 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:40.428471 12788 tablet_service.cc:1505] Processing CreateTablet for tablet 7d28e4999b504df4a53610187731416d (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260202 15:07:40.452296 12788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d28e4999b504df4a53610187731416d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.452147 12711 tablet_service.cc:1505] Processing CreateTablet for tablet d91ac50607b34cd1b7d61cd42f7c26d8 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260202 15:07:40.438326 12783 tablet_service.cc:1505] Processing CreateTablet for tablet 323d926990494ad1aafb6aa0282d9f68 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260202 15:07:40.453493 12783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323d926990494ad1aafb6aa0282d9f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.427628 12708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc895893bbe4569bb328cfc3513de38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.457108 12787 tablet_service.cc:1505] Processing CreateTablet for tablet c4798df17a1e4d18a594bb1b37b98bae (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260202 15:07:40.457403 12789 tablet_service.cc:1505] Processing CreateTablet for tablet 6fc895893bbe4569bb328cfc3513de38 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260202 15:07:40.458159 12787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4798df17a1e4d18a594bb1b37b98bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.458647 12789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc895893bbe4569bb328cfc3513de38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.459268 12786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435e192b63f348a58c00c3ae9e598ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.429054 12785 tablet_service.cc:1505] Processing CreateTablet for tablet 01f20099bc6844498a65c943dde6b1ff (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260202 15:07:40.460345 12785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f20099bc6844498a65c943dde6b1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.439469 12784 tablet_service.cc:1505] Processing CreateTablet for tablet 6f9321e2251943229a0b7dbdc0577c6b (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260202 15:07:40.467643 12784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9321e2251943229a0b7dbdc0577c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.461201 12631 tablet_service.cc:1505] Processing CreateTablet for tablet d91ac50607b34cd1b7d61cd42f7c26d8 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260202 15:07:40.471621 12707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2476071e1d24aa6a1342c8d81699cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.455446 12709 tablet_service.cc:1505] Processing CreateTablet for tablet 2ac5476ea30a46548db9ede417d946e6 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260202 15:07:40.476441 12709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac5476ea30a46548db9ede417d946e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.478835 12711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91ac50607b34cd1b7d61cd42f7c26d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.480526 12632 tablet_service.cc:1505] Processing CreateTablet for tablet 62adc5e06e9d428f94bdbd03c58c5e5b (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260202 15:07:40.443775 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87df3a9146bb41e1ac7468e17d2d9498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.491410 12842 tablet_bootstrap.cc:492] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:40.491811 12842 ts_tablet_manager.cc:1403] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.300s user 0.007s sys 0.014s
I20260202 15:07:40.421866 12841 raft_consensus.cc:359] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:40.492851 12841 raft_consensus.cc:385] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.493105 12841 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.493707 12841 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:40.495635 12841 ts_tablet_manager.cc:1434] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.078s user 0.004s sys 0.003s
I20260202 15:07:40.495523 12706 tablet_service.cc:1505] Processing CreateTablet for tablet 62adc5e06e9d428f94bdbd03c58c5e5b (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260202 15:07:40.496368 12841 tablet_bootstrap.cc:492] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:40.496569 12706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62adc5e06e9d428f94bdbd03c58c5e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.499500 12788 tablet_service.cc:1505] Processing CreateTablet for tablet d2476071e1d24aa6a1342c8d81699cef (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260202 15:07:40.500684 12788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2476071e1d24aa6a1342c8d81699cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.506762 12840 consensus_queue.cc:260] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.507097 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:40.507581 12852 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:40.494319 12842 raft_consensus.cc:359] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.513725 12842 raft_consensus.cc:385] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.521098 12842 raft_consensus.cc:740] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.522056 12842 consensus_queue.cc:260] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.513689 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:40.525959 12799 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:40.527575 12637 tablet_service.cc:1505] Processing CreateTablet for tablet 2ac5476ea30a46548db9ede417d946e6 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260202 15:07:40.528844 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91ac50607b34cd1b7d61cd42f7c26d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.534981 12637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac5476ea30a46548db9ede417d946e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.544731 12844 raft_consensus.cc:697] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:40.546228 12690 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:40.546425 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:40.511924 12852 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.546459 12844 consensus_queue.cc:237] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:40.556581 12688 leader_election.cc:343] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.557684 12712 tablet_service.cc:1505] Processing CreateTablet for tablet 488d33b86df249748c59fbf945f4e8b3 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260202 15:07:40.561558 12634 tablet_service.cc:1505] Processing CreateTablet for tablet 11a10c0c64914444a6659e661207c082 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260202 15:07:40.561820 12714 tablet_service.cc:1505] Processing CreateTablet for tablet 11a10c0c64914444a6659e661207c082 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260202 15:07:40.558789 12712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 488d33b86df249748c59fbf945f4e8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.567278 12714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11a10c0c64914444a6659e661207c082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.578789 12852 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:40.579612 12639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2476071e1d24aa6a1342c8d81699cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.580817 12638 tablet_service.cc:1505] Processing CreateTablet for tablet 488d33b86df249748c59fbf945f4e8b3 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260202 15:07:40.579440 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:40.585698 12634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11a10c0c64914444a6659e661207c082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.585707 12650 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 1: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:07:40.592826 12789 tablet_service.cc:1505] Processing CreateTablet for tablet d91ac50607b34cd1b7d61cd42f7c26d8 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260202 15:07:40.594144 12789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91ac50607b34cd1b7d61cd42f7c26d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.594805 12638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 488d33b86df249748c59fbf945f4e8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.583196 12852 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:40.595487 12852 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:40.595407 12784 tablet_service.cc:1505] Processing CreateTablet for tablet 2ac5476ea30a46548db9ede417d946e6 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260202 15:07:40.595916 12852 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:40.596452 12784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac5476ea30a46548db9ede417d946e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.604440 12841 tablet_bootstrap.cc:654] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.604200 12787 tablet_service.cc:1505] Processing CreateTablet for tablet 488d33b86df249748c59fbf945f4e8b3 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260202 15:07:40.605785 12787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 488d33b86df249748c59fbf945f4e8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.611233 12790 tablet_service.cc:1505] Processing CreateTablet for tablet 87df3a9146bb41e1ac7468e17d2d9498 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260202 15:07:40.618556 12633 tablet_service.cc:1505] Processing CreateTablet for tablet 21bc61b2f0f64d0b80e5d98615b1d6b6 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260202 15:07:40.624341 12842 ts_tablet_manager.cc:1434] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.132s user 0.015s sys 0.016s
I20260202 15:07:40.625308 12842 tablet_bootstrap.cc:492] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:40.625746 12716 tablet_service.cc:1505] Processing CreateTablet for tablet 21bc61b2f0f64d0b80e5d98615b1d6b6 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260202 15:07:40.626956 12716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bc61b2f0f64d0b80e5d98615b1d6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.627676 12840 ts_tablet_manager.cc:1434] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.212s user 0.022s sys 0.005s
I20260202 15:07:40.628458 12840 tablet_bootstrap.cc:492] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:40.632149 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62adc5e06e9d428f94bdbd03c58c5e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.634639 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bc61b2f0f64d0b80e5d98615b1d6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.636420 12783 tablet_service.cc:1505] Processing CreateTablet for tablet 62adc5e06e9d428f94bdbd03c58c5e5b (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260202 15:07:40.630767 12790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87df3a9146bb41e1ac7468e17d2d9498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.657490 12636 tablet_service.cc:1505] Processing CreateTablet for tablet da327288552c40f59790d9001c7f7e5d (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260202 15:07:40.658565 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da327288552c40f59790d9001c7f7e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.661099 12640 tablet_service.cc:1505] Processing CreateTablet for tablet af412f1b23854fdea68917d1df084a35 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260202 15:07:40.662143 12640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af412f1b23854fdea68917d1df084a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.668010 12786 tablet_service.cc:1505] Processing CreateTablet for tablet 11a10c0c64914444a6659e661207c082 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260202 15:07:40.673856 12782 tablet_service.cc:1505] Processing CreateTablet for tablet 21bc61b2f0f64d0b80e5d98615b1d6b6 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260202 15:07:40.674685 12783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62adc5e06e9d428f94bdbd03c58c5e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.676327 12782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bc61b2f0f64d0b80e5d98615b1d6b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260202 15:07:40.678097 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 126705us
I20260202 15:07:40.679486 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:40.681011 12690 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:40.679687 12840 tablet_bootstrap.cc:654] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.683836 12864 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:40.684913 12842 tablet_bootstrap.cc:654] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.685738 12781 tablet_service.cc:1505] Processing CreateTablet for tablet da327288552c40f59790d9001c7f7e5d (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260202 15:07:40.686790 12781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da327288552c40f59790d9001c7f7e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.687062 12708 tablet_service.cc:1505] Processing CreateTablet for tablet da327288552c40f59790d9001c7f7e5d (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260202 15:07:40.688131 12708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da327288552c40f59790d9001c7f7e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.689800 12864 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:40.692220 12853 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:40.692642 12853 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:40.694381 12853 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:40.696035 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:40.697769 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:40.700073 12615 leader_election.cc:343] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:40.702541 12615 leader_election.cc:343] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.703116 12615 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:40.705462 12853 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:40.711793 12841 tablet_bootstrap.cc:492] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:40.712235 12841 ts_tablet_manager.cc:1403] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.216s user 0.006s sys 0.006s
I20260202 15:07:40.715157 12841 raft_consensus.cc:359] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.715729 12841 raft_consensus.cc:385] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.715986 12841 raft_consensus.cc:740] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.716667 12841 consensus_queue.cc:260] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.720661 12841 ts_tablet_manager.cc:1434] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260202 15:07:40.721590 12841 tablet_bootstrap.cc:492] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:40.723357 12852 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.725723 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:40.726230 12650 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:40.727843 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:40.728296 12799 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:40.713544 12786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11a10c0c64914444a6659e661207c082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.726132 12852 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:40.718348 12710 tablet_service.cc:1505] Processing CreateTablet for tablet af412f1b23854fdea68917d1df084a35 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260202 15:07:40.738420 12615 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:40.739413 12853 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:40.740599 12710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af412f1b23854fdea68917d1df084a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.742463 12635 tablet_service.cc:1505] Processing CreateTablet for tablet fe02793fdaa448caa739180083258aae (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260202 15:07:40.743530 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe02793fdaa448caa739180083258aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.744347 12716 tablet_service.cc:1505] Processing CreateTablet for tablet fe02793fdaa448caa739180083258aae (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260202 15:07:40.744463 12546 catalog_manager.cc:5654] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:40.745388 12716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe02793fdaa448caa739180083258aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.753806 12784 tablet_service.cc:1505] Processing CreateTablet for tablet af412f1b23854fdea68917d1df084a35 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260202 15:07:40.755112 12650 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:40.756263 12688 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:40.756634 12789 tablet_service.cc:1505] Processing CreateTablet for tablet fe02793fdaa448caa739180083258aae (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260202 15:07:40.757004 12852 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:40.757454 12852 raft_consensus.cc:697] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:40.757701 12789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe02793fdaa448caa739180083258aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.760427 12841 tablet_bootstrap.cc:654] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.758177 12852 consensus_queue.cc:237] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.738435 12799 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:40.774811 12842 tablet_bootstrap.cc:492] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:40.774533 12640 tablet_service.cc:1505] Processing CreateTablet for tablet ed220e8ce3ce45ff8b531267a15de19e (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260202 15:07:40.775259 12842 ts_tablet_manager.cc:1403] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.150s user 0.014s sys 0.004s
I20260202 15:07:40.776082 12640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed220e8ce3ce45ff8b531267a15de19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.777817 12842 raft_consensus.cc:359] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.777747 12710 tablet_service.cc:1505] Processing CreateTablet for tablet ed220e8ce3ce45ff8b531267a15de19e (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260202 15:07:40.778486 12842 raft_consensus.cc:385] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.771610 12784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af412f1b23854fdea68917d1df084a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.779245 12710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed220e8ce3ce45ff8b531267a15de19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.779090 12787 tablet_service.cc:1505] Processing CreateTablet for tablet ed220e8ce3ce45ff8b531267a15de19e (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260202 15:07:40.781462 12709 tablet_service.cc:1505] Processing CreateTablet for tablet 965bcb05a43147b7be53449d0fe46509 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260202 15:07:40.778813 12842 raft_consensus.cc:740] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.789584 12787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed220e8ce3ce45ff8b531267a15de19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.789386 12842 consensus_queue.cc:260] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.791895 12842 ts_tablet_manager.cc:1434] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260202 15:07:40.792739 12842 tablet_bootstrap.cc:492] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:40.795523 12840 tablet_bootstrap.cc:492] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:40.796036 12840 ts_tablet_manager.cc:1403] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.168s user 0.005s sys 0.013s
I20260202 15:07:40.784539 12788 tablet_service.cc:1505] Processing CreateTablet for tablet 965bcb05a43147b7be53449d0fe46509 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260202 15:07:40.797086 12788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965bcb05a43147b7be53449d0fe46509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.797772 12842 tablet_bootstrap.cc:654] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.799108 12709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965bcb05a43147b7be53449d0fe46509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.808950 12631 tablet_service.cc:1505] Processing CreateTablet for tablet 965bcb05a43147b7be53449d0fe46509 (DEFAULT_TABLE table=test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260202 15:07:40.809154 12840 raft_consensus.cc:359] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.810065 12840 raft_consensus.cc:385] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.810611 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965bcb05a43147b7be53449d0fe46509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:40.810428 12840 raft_consensus.cc:740] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.815678 12841 tablet_bootstrap.cc:492] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:40.816231 12841 ts_tablet_manager.cc:1403] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.015s
I20260202 15:07:40.816533 12840 consensus_queue.cc:260] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.819494 12840 ts_tablet_manager.cc:1434] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.023s user 0.007s sys 0.001s
I20260202 15:07:40.820195 12841 raft_consensus.cc:359] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.820763 12840 tablet_bootstrap.cc:492] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:40.821122 12841 raft_consensus.cc:385] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.821516 12841 raft_consensus.cc:740] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.873014 12840 tablet_bootstrap.cc:654] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.859577 12841 consensus_queue.cc:260] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.876668 12841 ts_tablet_manager.cc:1434] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.060s user 0.007s sys 0.001s
I20260202 15:07:40.877486 12841 tablet_bootstrap.cc:492] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:40.883471 12842 tablet_bootstrap.cc:492] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:40.883940 12842 ts_tablet_manager.cc:1403] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.091s user 0.021s sys 0.037s
I20260202 15:07:40.886941 12840 tablet_bootstrap.cc:492] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:40.887362 12840 ts_tablet_manager.cc:1403] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.004s
I20260202 15:07:40.886765 12842 raft_consensus.cc:359] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:40.887717 12596 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:07:40.887827 12842 raft_consensus.cc:385] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.888206 12842 raft_consensus.cc:740] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.888935 12842 consensus_queue.cc:260] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.890223 12840 raft_consensus.cc:359] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.890920 12840 raft_consensus.cc:385] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.891180 12840 raft_consensus.cc:740] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.891884 12840 consensus_queue.cc:260] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.893307 12841 tablet_bootstrap.cc:654] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.893833 12840 ts_tablet_manager.cc:1434] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260202 15:07:40.894665 12840 tablet_bootstrap.cc:492] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:40.896920 12842 ts_tablet_manager.cc:1434] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20260202 15:07:40.898095 12842 tablet_bootstrap.cc:492] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:40.899924 12840 tablet_bootstrap.cc:654] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.901845 12841 tablet_bootstrap.cc:492] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:40.902294 12841 ts_tablet_manager.cc:1403] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.001s
I20260202 15:07:40.904484 12842 tablet_bootstrap.cc:654] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.905063 12841 raft_consensus.cc:359] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.905809 12841 raft_consensus.cc:385] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.906229 12841 raft_consensus.cc:740] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.907172 12852 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:40.907114 12841 consensus_queue.cc:260] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.907698 12852 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.909474 12852 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:40.910372 12841 ts_tablet_manager.cc:1434] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260202 15:07:40.911587 12841 tablet_bootstrap.cc:492] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:40.912024 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:40.913017 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:40.913503 12650 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
W20260202 15:07:40.914510 12690 leader_election.cc:343] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.916594 12688 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:40.917747 12545 catalog_manager.cc:5654] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:40.918531 12841 tablet_bootstrap.cc:654] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.920636 12852 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:40.921056 12842 tablet_bootstrap.cc:492] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:40.921016 12852 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:40.921435 12852 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:40.923020 12842 ts_tablet_manager.cc:1403] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.016s
I20260202 15:07:40.924393 12841 tablet_bootstrap.cc:492] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:40.924749 12841 ts_tablet_manager.cc:1403] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260202 15:07:40.925416 12842 raft_consensus.cc:359] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.925985 12842 raft_consensus.cc:385] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.926221 12842 raft_consensus.cc:740] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.926810 12842 consensus_queue.cc:260] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.927044 12841 raft_consensus.cc:359] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.927737 12841 raft_consensus.cc:385] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.927675 12852 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.928118 12841 raft_consensus.cc:740] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.929188 12842 ts_tablet_manager.cc:1434] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:07:40.928779 12841 consensus_queue.cc:260] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.930181 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:40.930740 12650 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:40.931170 12841 ts_tablet_manager.cc:1434] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260202 15:07:40.932037 12841 tablet_bootstrap.cc:492] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:40.936677 12650 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:40.937213 12841 tablet_bootstrap.cc:654] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.937567 12688 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:40.938385 12866 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:40.938814 12866 raft_consensus.cc:697] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:40.939585 12866 consensus_queue.cc:237] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.940338 12852 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:40.939960 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:07:40.943202 12690 leader_election.cc:343] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.943534 12841 tablet_bootstrap.cc:492] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:40.943997 12841 ts_tablet_manager.cc:1403] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260202 15:07:40.930372 12842 tablet_bootstrap.cc:492] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:40.946460 12841 raft_consensus.cc:359] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.947108 12841 raft_consensus.cc:385] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.947048 12844 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:40.947543 12844 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.949143 12844 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:40.950516 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:40.951252 12842 tablet_bootstrap.cc:654] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.951824 12840 tablet_bootstrap.cc:492] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
W20260202 15:07:40.951777 12765 leader_election.cc:343] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.952316 12840 ts_tablet_manager.cc:1403] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.002s
I20260202 15:07:40.954974 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:40.954928 12840 raft_consensus.cc:359] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.955868 12842 tablet_bootstrap.cc:492] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:40.955961 12841 raft_consensus.cc:740] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
W20260202 15:07:40.955967 12763 leader_election.cc:343] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:40.956431 12842 ts_tablet_manager.cc:1403] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.003s
I20260202 15:07:40.956506 12763 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:40.956727 12841 consensus_queue.cc:260] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.957418 12844 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:40.955602 12840 raft_consensus.cc:385] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.957316 12547 catalog_manager.cc:5654] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:40.958019 12840 raft_consensus.cc:740] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.959008 12841 ts_tablet_manager.cc:1434] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260202 15:07:40.958719 12840 consensus_queue.cc:260] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.959136 12842 raft_consensus.cc:359] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:40.960023 12842 raft_consensus.cc:385] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.960371 12842 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.961273 12840 ts_tablet_manager.cc:1434] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260202 15:07:40.961072 12842 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:40.961727 12841 tablet_bootstrap.cc:492] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:40.965767 12842 ts_tablet_manager.cc:1434] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20260202 15:07:40.966557 12842 tablet_bootstrap.cc:492] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:40.967392 12841 tablet_bootstrap.cc:654] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.971890 12842 tablet_bootstrap.cc:654] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.978853 12840 tablet_bootstrap.cc:492] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:40.979770 12842 tablet_bootstrap.cc:492] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:40.980165 12842 ts_tablet_manager.cc:1403] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260202 15:07:40.981329 12841 tablet_bootstrap.cc:492] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:40.981688 12841 ts_tablet_manager.cc:1403] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.010s
I20260202 15:07:40.981812 12842 raft_consensus.cc:359] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.982280 12842 raft_consensus.cc:385] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.982460 12842 raft_consensus.cc:740] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.983002 12842 consensus_queue.cc:260] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.983858 12841 raft_consensus.cc:359] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.984433 12841 raft_consensus.cc:385] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:40.984678 12841 raft_consensus.cc:740] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:40.984853 12842 ts_tablet_manager.cc:1434] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260202 15:07:40.985651 12842 tablet_bootstrap.cc:492] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:40.985371 12841 consensus_queue.cc:260] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:40.986480 12840 tablet_bootstrap.cc:654] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.990043 12841 ts_tablet_manager.cc:1434] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260202 15:07:40.990993 12842 tablet_bootstrap.cc:654] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:40.991022 12841 tablet_bootstrap.cc:492] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:40.995877 12841 tablet_bootstrap.cc:654] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.002395 12840 tablet_bootstrap.cc:492] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.002599 12844 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.002763 12842 tablet_bootstrap.cc:492] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.002804 12840 ts_tablet_manager.cc:1403] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.013s
I20260202 15:07:41.003221 12842 ts_tablet_manager.cc:1403] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20260202 15:07:41.003023 12844 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.003767 12841 tablet_bootstrap.cc:492] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.004136 12841 ts_tablet_manager.cc:1403] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260202 15:07:41.005297 12840 raft_consensus.cc:359] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:41.005435 12842 raft_consensus.cc:359] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.005944 12840 raft_consensus.cc:385] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.006044 12842 raft_consensus.cc:385] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.006145 12841 raft_consensus.cc:359] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.006301 12840 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.006742 12841 raft_consensus.cc:385] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.006958 12841 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.004714 12844 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:41.007073 12840 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:41.007405 12841 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.007737 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.009032 12840 ts_tablet_manager.cc:1434] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:07:41.009409 12841 ts_tablet_manager.cc:1434] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260202 15:07:41.009922 12840 tablet_bootstrap.cc:492] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.010234 12841 tablet_bootstrap.cc:492] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.006371 12842 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.006240 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.011024 12726 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:41.011112 12842 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.012012 12765 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:41.012913 12842 ts_tablet_manager.cc:1434] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.009s user 0.003s sys 0.001s
I20260202 15:07:41.013034 12865 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.013451 12865 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.013774 12842 tablet_bootstrap.cc:492] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
W20260202 15:07:41.014140 12763 leader_election.cc:343] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.013772 12865 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.015131 12841 tablet_bootstrap.cc:654] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.018757 12842 tablet_bootstrap.cc:654] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.025527 12840 tablet_bootstrap.cc:654] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.025979 12841 tablet_bootstrap.cc:492] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.026432 12841 ts_tablet_manager.cc:1403] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.004s
I20260202 15:07:41.027261 12865 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.028700 12841 raft_consensus.cc:359] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.029317 12841 raft_consensus.cc:385] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.029541 12842 tablet_bootstrap.cc:492] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.029590 12841 raft_consensus.cc:740] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.029728 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:07:41.030964 12763 leader_election.cc:343] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.030555 12841 consensus_queue.cc:260] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.032652 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.033125 12726 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.033259 12841 ts_tablet_manager.cc:1434] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260202 15:07:41.034511 12841 tablet_bootstrap.cc:492] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.039752 12726 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:41.040865 12765 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:41.041129 12841 tablet_bootstrap.cc:654] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.041678 12844 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.042160 12844 raft_consensus.cc:697] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:41.042289 12853 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.042634 12853 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.043037 12844 consensus_queue.cc:237] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.044517 12853 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.045625 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.046154 12865 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:41.030007 12842 ts_tablet_manager.cc:1403] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260202 15:07:41.046849 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:41.047612 12615 leader_election.cc:343] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:41.048442 12615 leader_election.cc:343] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.048940 12615 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:41.048992 12842 raft_consensus.cc:359] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.049870 12842 raft_consensus.cc:385] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.050205 12842 raft_consensus.cc:740] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.050477 12853 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:41.050967 12842 consensus_queue.cc:260] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.053915 12842 ts_tablet_manager.cc:1434] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260202 15:07:41.054802 12842 tablet_bootstrap.cc:492] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.053683 12865 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.057163 12865 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.058327 12863 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.058746 12863 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:41.060572 12863 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:07:41.063351 12842 tablet_bootstrap.cc:654] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.063423 12840 tablet_bootstrap.cc:492] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.064054 12840 ts_tablet_manager.cc:1403] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.054s user 0.014s sys 0.017s
I20260202 15:07:41.063788 12546 catalog_manager.cc:5654] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:41.067790 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.068518 12725 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:41.070802 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.060588 12865 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:41.077739 12765 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:41.080615 12853 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.081243 12844 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.081140 12853 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.081561 12844 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.081893 12844 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.083211 12853 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.085215 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.085745 12799 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.086843 12615 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.087121 12840 raft_consensus.cc:359] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.087783 12840 raft_consensus.cc:385] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.087879 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.088141 12840 raft_consensus.cc:740] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.088716 12853 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.089215 12853 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.088900 12840 consensus_queue.cc:260] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.089718 12853 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.090165 12726 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.091444 12840 ts_tablet_manager.cc:1434] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.027s user 0.005s sys 0.003s
I20260202 15:07:41.094375 12844 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:41.095012 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.095841 12649 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:41.094340 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.099442 12650 raft_consensus.cc:2393] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 1: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:07:41.101065 12853 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.101660 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.102259 12726 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.103168 12853 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.104326 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.104858 12725 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.105170 12841 tablet_bootstrap.cc:492] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.105612 12841 ts_tablet_manager.cc:1403] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.071s user 0.016s sys 0.022s
I20260202 15:07:41.106992 12844 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:07:41.107134 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.107846 12799 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.108273 12841 raft_consensus.cc:359] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.108840 12840 tablet_bootstrap.cc:492] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.108903 12841 raft_consensus.cc:385] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.109354 12842 tablet_bootstrap.cc:492] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.109319 12841 raft_consensus.cc:740] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.109859 12842 ts_tablet_manager.cc:1403] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.055s user 0.020s sys 0.014s
I20260202 15:07:41.110226 12841 consensus_queue.cc:260] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.114950 12799 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.115806 12840 tablet_bootstrap.cc:654] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.116086 12615 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.117007 12853 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.117368 12726 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:41.112288 12842 raft_consensus.cc:359] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.118809 12842 raft_consensus.cc:385] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.119267 12842 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.118728 12765 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:41.119879 12842 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.120373 12844 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.121204 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:41.112303 12841 ts_tablet_manager.cc:1434] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:07:41.121868 12648 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.121963 12844 raft_consensus.cc:697] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:41.123507 12844 consensus_queue.cc:237] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } }
I20260202 15:07:41.125548 12841 tablet_bootstrap.cc:492] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.126040 12725 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.121968 12842 ts_tablet_manager.cc:1434] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20260202 15:07:41.130765 12842 tablet_bootstrap.cc:492] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.134604 12648 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:41.135574 12853 raft_consensus.cc:697] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:41.137149 12763 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:41.139464 12853 consensus_queue.cc:237] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.140985 12896 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.141431 12844 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.136540 12841 tablet_bootstrap.cc:654] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.137876 12842 tablet_bootstrap.cc:654] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.154659 12547 catalog_manager.cc:5654] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.155661 12898 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.156056 12898 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.157817 12898 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.158306 12896 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:41.159874 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.160477 12799 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.161387 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.162955 12615 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.162966 12898 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.164415 12898 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.167740 12841 tablet_bootstrap.cc:492] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.168159 12841 ts_tablet_manager.cc:1403] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.001s
I20260202 15:07:41.168713 12866 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.170457 12841 raft_consensus.cc:359] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.171144 12841 raft_consensus.cc:385] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.171375 12841 raft_consensus.cc:740] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.172437 12841 consensus_queue.cc:260] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.172807 12906 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.173105 12840 tablet_bootstrap.cc:492] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.173159 12906 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.173554 12840 ts_tablet_manager.cc:1403] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.007s
I20260202 15:07:41.173533 12906 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.176029 12840 raft_consensus.cc:359] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.176576 12840 raft_consensus.cc:385] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.176874 12840 raft_consensus.cc:740] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.172907 12898 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.182327 12840 consensus_queue.cc:260] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.183409 12841 ts_tablet_manager.cc:1434] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.015s user 0.005s sys 0.011s
I20260202 15:07:41.184419 12841 tablet_bootstrap.cc:492] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.185954 12840 ts_tablet_manager.cc:1434] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260202 15:07:41.188153 12840 tablet_bootstrap.cc:492] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.190804 12841 tablet_bootstrap.cc:654] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.201231 12844 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.202572 12906 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.204139 12866 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260202 15:07:41.213959 12866 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.214380 12866 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.215251 12840 tablet_bootstrap.cc:654] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.216950 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.217499 12649 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:41.218807 12688 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.219815 12864 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.220160 12864 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.220492 12864 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.231943 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.232620 12799 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:41.233952 12866 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.235417 12864 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.238183 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:41.238768 12649 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:41.244530 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 81653.6us
W20260202 15:07:41.245450 12615 leader_election.cc:343] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.247900 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.249979 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.250650 12799 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.254231 12905 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.256518 12905 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.257476 12842 tablet_bootstrap.cc:492] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.257875 12842 ts_tablet_manager.cc:1403] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.127s user 0.021s sys 0.025s
I20260202 15:07:41.258984 12905 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.260237 12842 raft_consensus.cc:359] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.277765 12842 raft_consensus.cc:385] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.278017 12842 raft_consensus.cc:740] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.278568 12841 tablet_bootstrap.cc:492] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.279065 12841 ts_tablet_manager.cc:1403] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.095s user 0.047s sys 0.028s
I20260202 15:07:41.278582 12842 consensus_queue.cc:260] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.279160 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.279953 12797 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.280684 12842 ts_tablet_manager.cc:1434] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260202 15:07:41.281474 12842 tablet_bootstrap.cc:492] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.262989 12864 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.283051 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.283520 12800 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.266055 12649 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:41.273185 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.285554 12726 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.261309 12547 catalog_manager.cc:5654] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.273810 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.287032 12842 tablet_bootstrap.cc:654] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:41.273422 12615 leader_election.cc:343] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.274456 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.281474 12841 raft_consensus.cc:359] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.291616 12841 raft_consensus.cc:385] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.291882 12841 raft_consensus.cc:740] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.292419 12841 consensus_queue.cc:260] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.294312 12841 ts_tablet_manager.cc:1434] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20260202 15:07:41.295161 12615 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:07:41.295928 12615 leader_election.cc:343] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.296676 12853 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.297082 12853 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.207996 12906 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.297454 12853 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.297932 12726 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
W20260202 15:07:41.298758 12615 leader_election.cc:343] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.299256 12841 tablet_bootstrap.cc:492] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.299175 12615 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:41.300005 12905 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:41.301187 12615 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.301831 12905 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.302282 12905 raft_consensus.cc:697] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:41.308717 12905 consensus_queue.cc:237] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.305224 12841 tablet_bootstrap.cc:654] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.315747 12799 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:41.307255 12688 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.316960 12800 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.317754 12915 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.320634 12915 raft_consensus.cc:697] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:41.321684 12915 consensus_queue.cc:237] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.324548 12840 tablet_bootstrap.cc:492] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.324949 12840 ts_tablet_manager.cc:1403] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.137s user 0.016s sys 0.019s
I20260202 15:07:41.318053 12841 tablet_bootstrap.cc:492] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.319415 12853 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.327879 12842 tablet_bootstrap.cc:492] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.328282 12842 ts_tablet_manager.cc:1403] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.027s
I20260202 15:07:41.328886 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.329128 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.329649 12800 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.330492 12842 raft_consensus.cc:359] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.331144 12842 raft_consensus.cc:385] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.331393 12842 raft_consensus.cc:740] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.326630 12840 raft_consensus.cc:359] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.332317 12840 raft_consensus.cc:385] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.332554 12840 raft_consensus.cc:740] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.333153 12840 consensus_queue.cc:260] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.333262 12547 catalog_manager.cc:5654] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.334928 12842 consensus_queue.cc:260] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.335386 12800 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.336275 12615 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.336896 12853 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.337071 12726 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.337368 12853 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.337666 12840 ts_tablet_manager.cc:1434] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260202 15:07:41.337740 12853 raft_consensus.cc:697] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:41.338376 12840 tablet_bootstrap.cc:492] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.338416 12853 consensus_queue.cc:237] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.326521 12841 ts_tablet_manager.cc:1403] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20260202 15:07:41.342046 12841 raft_consensus.cc:359] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.344213 12841 raft_consensus.cc:385] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.344424 12841 raft_consensus.cc:740] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.345149 12840 tablet_bootstrap.cc:654] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.345029 12841 consensus_queue.cc:260] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.345569 12546 catalog_manager.cc:5654] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.346879 12841 ts_tablet_manager.cc:1434] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260202 15:07:41.347648 12841 tablet_bootstrap.cc:492] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.352219 12726 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.337203 12842 ts_tablet_manager.cc:1434] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.009s user 0.001s sys 0.005s
I20260202 15:07:41.356570 12840 tablet_bootstrap.cc:492] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.357087 12840 ts_tablet_manager.cc:1403] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20260202 15:07:41.360131 12842 tablet_bootstrap.cc:492] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.359784 12840 raft_consensus.cc:359] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.360572 12840 raft_consensus.cc:385] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.360941 12840 raft_consensus.cc:740] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.361588 12840 consensus_queue.cc:260] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.363467 12841 tablet_bootstrap.cc:654] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.363794 12840 ts_tablet_manager.cc:1434] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260202 15:07:41.364557 12840 tablet_bootstrap.cc:492] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.369366 12840 tablet_bootstrap.cc:654] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.376999 12547 catalog_manager.cc:5654] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.380924 12842 tablet_bootstrap.cc:654] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.387763 12841 tablet_bootstrap.cc:492] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.388335 12841 ts_tablet_manager.cc:1403] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.008s
I20260202 15:07:41.390803 12841 raft_consensus.cc:359] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.391414 12841 raft_consensus.cc:385] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.391716 12840 tablet_bootstrap.cc:492] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.391646 12841 raft_consensus.cc:740] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.391878 12842 tablet_bootstrap.cc:492] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.392113 12840 ts_tablet_manager.cc:1403] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.012s
I20260202 15:07:41.392244 12842 ts_tablet_manager.cc:1403] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.005s
I20260202 15:07:41.392292 12841 consensus_queue.cc:260] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.394362 12841 ts_tablet_manager.cc:1434] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:07:41.394294 12840 raft_consensus.cc:359] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.394616 12842 raft_consensus.cc:359] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.394984 12840 raft_consensus.cc:385] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.395239 12842 raft_consensus.cc:385] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.395305 12840 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.395573 12841 tablet_bootstrap.cc:492] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.395524 12842 raft_consensus.cc:740] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.396028 12840 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.396344 12842 consensus_queue.cc:260] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.401319 12841 tablet_bootstrap.cc:654] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.401464 12840 ts_tablet_manager.cc:1434] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260202 15:07:41.402216 12840 tablet_bootstrap.cc:492] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.406728 12853 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.407282 12842 ts_tablet_manager.cc:1434] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260202 15:07:41.407166 12853 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.407905 12840 tablet_bootstrap.cc:654] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.408897 12853 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.410473 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.410808 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.411176 12726 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.411577 12800 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.412232 12615 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.412547 12841 tablet_bootstrap.cc:492] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.413002 12841 ts_tablet_manager.cc:1403] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20260202 15:07:41.417769 12853 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.418123 12853 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.418426 12853 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.422377 12841 raft_consensus.cc:359] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.423388 12841 raft_consensus.cc:385] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.423768 12841 raft_consensus.cc:740] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.423878 12842 tablet_bootstrap.cc:492] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.424566 12841 consensus_queue.cc:260] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.425082 12853 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.427459 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.427942 12726 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.431177 12842 tablet_bootstrap.cc:654] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.434370 12841 ts_tablet_manager.cc:1434] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20260202 15:07:41.435241 12841 tablet_bootstrap.cc:492] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.442047 12842 tablet_bootstrap.cc:492] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.435382 12840 tablet_bootstrap.cc:492] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.435461 12726 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.436290 12853 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.437529 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.442843 12840 ts_tablet_manager.cc:1403] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.023s
I20260202 15:07:41.443027 12800 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.443496 12615 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.444104 12853 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.444475 12853 raft_consensus.cc:697] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:41.445089 12840 raft_consensus.cc:359] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.445184 12853 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.445652 12840 raft_consensus.cc:385] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.445991 12840 raft_consensus.cc:740] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.446587 12840 consensus_queue.cc:260] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.448597 12841 tablet_bootstrap.cc:654] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.448734 12840 ts_tablet_manager.cc:1434] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260202 15:07:41.449836 12840 tablet_bootstrap.cc:492] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.442461 12842 ts_tablet_manager.cc:1403] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.002s
I20260202 15:07:41.452836 12842 raft_consensus.cc:359] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.453369 12842 raft_consensus.cc:385] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.453583 12842 raft_consensus.cc:740] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.454167 12842 consensus_queue.cc:260] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.457129 12800 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.459748 12844 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.460109 12842 ts_tablet_manager.cc:1434] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260202 15:07:41.460242 12844 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.460958 12842 tablet_bootstrap.cc:492] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.462150 12844 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:41.462451 12840 tablet_bootstrap.cc:654] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.463518 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.467190 12842 tablet_bootstrap.cc:654] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.466485 12546 catalog_manager.cc:5654] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.469318 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:07:41.471498 12765 leader_election.cc:343] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:41.473079 12763 leader_election.cc:343] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.473459 12763 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:41.474079 12896 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.474205 12840 tablet_bootstrap.cc:492] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.474578 12840 ts_tablet_manager.cc:1403] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.000s
I20260202 15:07:41.474474 12896 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.475288 12844 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:41.476238 12896 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:41.474784 12853 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.476804 12853 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.477257 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:07:41.478907 12765 leader_election.cc:343] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.479583 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.480317 12649 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:41.481632 12840 raft_consensus.cc:359] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.482187 12840 raft_consensus.cc:385] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.482160 12915 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:41.482611 12840 raft_consensus.cc:740] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.482111 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.483230 12840 consensus_queue.cc:260] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.483954 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:41.484097 12615 leader_election.cc:343] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.478648 12853 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.485047 12840 ts_tablet_manager.cc:1434] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
W20260202 15:07:41.485036 12615 leader_election.cc:343] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.485601 12615 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:41.485926 12840 tablet_bootstrap.cc:492] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.486810 12853 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:41.487426 12763 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:41.488355 12896 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.488691 12896 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.488986 12896 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.490897 12840 tablet_bootstrap.cc:654] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.495081 12840 tablet_bootstrap.cc:492] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.495517 12840 ts_tablet_manager.cc:1403] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260202 15:07:41.497668 12840 raft_consensus.cc:359] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.498207 12840 raft_consensus.cc:385] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.498433 12840 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.498950 12840 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.502715 12840 ts_tablet_manager.cc:1434] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260202 15:07:41.503103 12841 tablet_bootstrap.cc:492] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.502943 12896 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.503454 12840 tablet_bootstrap.cc:492] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.503585 12841 ts_tablet_manager.cc:1403] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.069s user 0.015s sys 0.005s
I20260202 15:07:41.505358 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.505843 12726 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.505901 12841 raft_consensus.cc:359] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.506716 12841 raft_consensus.cc:385] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.507004 12841 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.507334 12852 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.507624 12841 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.508538 12840 tablet_bootstrap.cc:654] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.510290 12865 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.510751 12865 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.511993 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:41.512382 12865 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:41.512488 12650 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.513218 12841 ts_tablet_manager.cc:1434] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260202 15:07:41.513728 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.514166 12841 tablet_bootstrap.cc:492] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.514273 12724 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:41.515304 12765 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:41.515993 12865 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.516290 12865 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.516557 12865 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.518374 12650 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:41.519381 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.519591 12763 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:07:41.520430 12763 leader_election.cc:343] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.520649 12841 tablet_bootstrap.cc:654] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.527910 12840 tablet_bootstrap.cc:492] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.528265 12840 ts_tablet_manager.cc:1403] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.003s
I20260202 15:07:41.529363 12726 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:41.530483 12840 raft_consensus.cc:359] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.531124 12840 raft_consensus.cc:385] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.531360 12840 raft_consensus.cc:740] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.531858 12840 consensus_queue.cc:260] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.533921 12840 ts_tablet_manager.cc:1434] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260202 15:07:41.533841 12844 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.534587 12844 raft_consensus.cc:697] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:41.534850 12840 tablet_bootstrap.cc:492] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.534607 12865 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.535578 12844 consensus_queue.cc:237] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.537168 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:07:41.538319 12763 leader_election.cc:343] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.539845 12853 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.539995 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.540546 12840 tablet_bootstrap.cc:654] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.540596 12726 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.540501 12853 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.542913 12853 leader_election.cc:290] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.544356 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.544893 12799 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.546015 12615 leader_election.cc:304] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.547171 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:07:41.548847 12615 leader_election.cc:343] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.549630 12865 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:41.549773 12906 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.544009 12896 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:41.550304 12906 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.550907 12906 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.558154 12853 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.559803 12726 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:41.561264 12765 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:41.562924 12896 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.563553 12896 raft_consensus.cc:697] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:41.564488 12896 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.560287 12906 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.568766 12842 tablet_bootstrap.cc:492] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.569217 12906 leader_election.cc:290] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.573227 12863 consensus_queue.cc:1048] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.575580 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.576043 12799 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.579221 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:07:41.580541 12615 leader_election.cc:343] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.582154 12546 catalog_manager.cc:5654] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:41.584441 12799 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.585451 12615 leader_election.cc:304] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:41.586004 12906 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.586382 12906 raft_consensus.cc:697] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:41.569264 12842 ts_tablet_manager.cc:1403] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.108s user 0.010s sys 0.001s
I20260202 15:07:41.587067 12906 consensus_queue.cc:237] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.589396 12842 raft_consensus.cc:359] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.590240 12842 raft_consensus.cc:385] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.597569 12853 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.605377 12544 catalog_manager.cc:5654] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.609522 12841 tablet_bootstrap.cc:492] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.611303 12841 ts_tablet_manager.cc:1403] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.097s user 0.030s sys 0.005s
I20260202 15:07:41.613503 12841 raft_consensus.cc:359] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.613857 12649 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.614202 12841 raft_consensus.cc:385] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.614512 12841 raft_consensus.cc:740] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.615124 12841 consensus_queue.cc:260] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.617259 12841 ts_tablet_manager.cc:1434] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:07:41.618076 12841 tablet_bootstrap.cc:492] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.590574 12842 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.621202 12842 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.623203 12841 tablet_bootstrap.cc:654] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.623663 12906 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.624111 12906 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.625672 12906 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.628190 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.628911 12800 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.629721 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.629909 12615 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.637696 12863 consensus_queue.cc:1048] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:07:41.640429 12615 leader_election.cc:343] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.642575 12906 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.642922 12906 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.643213 12906 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.644510 12853 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.645066 12853 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.648633 12842 ts_tablet_manager.cc:1434] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20260202 15:07:41.649049 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.651882 12726 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.649436 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.652988 12615 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:07:41.653784 12615 leader_election.cc:343] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.649400 12906 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.649935 12853 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.657619 12841 tablet_bootstrap.cc:492] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.657960 12841 ts_tablet_manager.cc:1403] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.008s
I20260202 15:07:41.651424 12842 tablet_bootstrap.cc:492] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.654277 12840 tablet_bootstrap.cc:492] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.654754 12907 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.655395 12545 catalog_manager.cc:5654] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:41.660491 12907 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.660457 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.660998 12907 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.661190 12800 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.666038 12842 tablet_bootstrap.cc:654] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.667469 12800 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.659475 12906 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.659816 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.670346 12841 raft_consensus.cc:359] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.671027 12841 raft_consensus.cc:385] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.671270 12841 raft_consensus.cc:740] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.667589 12907 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.671880 12841 consensus_queue.cc:260] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.673008 12907 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.674320 12841 ts_tablet_manager.cc:1434] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260202 15:07:41.675127 12841 tablet_bootstrap.cc:492] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.675244 12615 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:07:41.676352 12615 leader_election.cc:343] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.678392 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.679059 12726 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.681671 12841 tablet_bootstrap.cc:654] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.681732 12852 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.682159 12852 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.682552 12842 tablet_bootstrap.cc:492] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.683036 12842 ts_tablet_manager.cc:1403] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.012s
I20260202 15:07:41.685659 12852 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.686352 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.686834 12650 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:41.687678 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.687817 12688 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.687885 12842 raft_consensus.cc:359] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.688302 12726 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.688529 12842 raft_consensus.cc:385] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.688573 12852 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.688860 12842 raft_consensus.cc:740] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.689029 12852 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:07:41.689128 12690 leader_election.cc:343] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.689257 12615 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.689459 12852 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.689522 12842 consensus_queue.cc:260] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.690131 12906 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.690172 12907 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.690747 12906 raft_consensus.cc:697] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:41.690908 12907 raft_consensus.cc:697] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:41.691596 12906 consensus_queue.cc:237] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.691700 12907 consensus_queue.cc:237] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.692589 12842 ts_tablet_manager.cc:1434] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260202 15:07:41.668390 12840 ts_tablet_manager.cc:1403] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.134s user 0.035s sys 0.070s
I20260202 15:07:41.652808 12896 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260202 15:07:41.693378 12842 tablet_bootstrap.cc:492] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.696239 12840 raft_consensus.cc:359] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.696807 12840 raft_consensus.cc:385] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.697065 12840 raft_consensus.cc:740] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.698810 12865 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.698913 12840 consensus_queue.cc:260] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.707396 12840 ts_tablet_manager.cc:1434] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260202 15:07:41.707980 12840 tablet_bootstrap.cc:492] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.711778 12864 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.715627 12864 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.717799 12864 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.705279 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.719646 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.720461 12649 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
W20260202 15:07:41.722471 12615 leader_election.cc:343] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.728114 12688 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.729382 12864 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.729835 12864 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.730407 12864 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.734292 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:41.736042 12690 leader_election.cc:343] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.745344 12841 tablet_bootstrap.cc:492] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.745954 12841 ts_tablet_manager.cc:1403] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.006s
I20260202 15:07:41.756436 12841 raft_consensus.cc:359] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.756851 12841 raft_consensus.cc:385] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.757056 12841 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.757723 12841 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.747282 12852 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.761147 12841 ts_tablet_manager.cc:1434] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.015s user 0.003s sys 0.002s
I20260202 15:07:41.761312 12852 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.761905 12841 tablet_bootstrap.cc:492] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.761929 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.762249 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:41.762864 12650 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:41.763170 12690 leader_election.cc:343] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.771664 12864 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.773216 12650 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:41.771298 12546 catalog_manager.cc:5654] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.774235 12688 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:41.774870 12852 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.775462 12842 tablet_bootstrap.cc:654] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.774842 12546 catalog_manager.cc:5654] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.776607 12841 tablet_bootstrap.cc:654] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.776002 12852 raft_consensus.cc:697] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:41.778347 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:41.778311 12852 consensus_queue.cc:237] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.779232 12650 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.781653 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:07:41.782907 12690 leader_election.cc:343] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.784165 12864 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.796801 12650 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:41.797721 12688 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:41.799769 12840 tablet_bootstrap.cc:654] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.802896 12852 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.805176 12842 tablet_bootstrap.cc:492] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.805581 12842 ts_tablet_manager.cc:1403] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.112s user 0.006s sys 0.019s
I20260202 15:07:41.807802 12842 raft_consensus.cc:359] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.808370 12842 raft_consensus.cc:385] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.808506 12841 tablet_bootstrap.cc:492] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.808622 12842 raft_consensus.cc:740] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.808877 12841 ts_tablet_manager.cc:1403] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.000s
I20260202 15:07:41.809443 12864 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.809921 12864 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.811775 12864 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.811540 12544 catalog_manager.cc:5654] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.813750 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.814754 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:41.815029 12688 leader_election.cc:343] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:41.815879 12690 leader_election.cc:343] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.816267 12690 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:41.816970 12864 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:41.817414 12852 raft_consensus.cc:697] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:41.818260 12852 consensus_queue.cc:237] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.820217 12840 tablet_bootstrap.cc:492] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.820606 12840 ts_tablet_manager.cc:1403] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.014s
I20260202 15:07:41.821648 12842 consensus_queue.cc:260] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.819352 12841 raft_consensus.cc:359] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.823040 12841 raft_consensus.cc:385] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.823318 12841 raft_consensus.cc:740] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.823288 12840 raft_consensus.cc:359] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.824113 12840 raft_consensus.cc:385] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.824476 12840 raft_consensus.cc:740] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.824165 12841 consensus_queue.cc:260] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.825045 12840 consensus_queue.cc:260] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.827268 12840 ts_tablet_manager.cc:1434] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260202 15:07:41.827545 12842 ts_tablet_manager.cc:1434] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260202 15:07:41.828008 12840 tablet_bootstrap.cc:492] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.828311 12842 tablet_bootstrap.cc:492] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.833858 12840 tablet_bootstrap.cc:654] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.835796 12907 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.836205 12907 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.837853 12907 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.838255 12841 ts_tablet_manager.cc:1434] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.029s user 0.005s sys 0.002s
I20260202 15:07:41.839172 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.839545 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.840104 12800 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:41.840389 12842 tablet_bootstrap.cc:654] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.839704 12544 catalog_manager.cc:5654] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
W20260202 15:07:41.841259 12615 leader_election.cc:343] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.842149 12615 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:41.843039 12907 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.843364 12907 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.843674 12907 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.848403 12840 tablet_bootstrap.cc:492] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.848760 12840 ts_tablet_manager.cc:1403] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.008s
I20260202 15:07:41.849428 12907 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.850514 12906 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.851207 12907 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.851011 12840 raft_consensus.cc:359] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.851622 12840 raft_consensus.cc:385] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.851843 12840 raft_consensus.cc:740] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.852421 12840 consensus_queue.cc:260] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.852752 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:41.854370 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:41.854830 12800 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.855290 12841 tablet_bootstrap.cc:492] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.859393 12842 tablet_bootstrap.cc:492] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.859790 12842 ts_tablet_manager.cc:1403] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.002s
I20260202 15:07:41.861624 12841 tablet_bootstrap.cc:654] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.862392 12842 raft_consensus.cc:359] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.863055 12842 raft_consensus.cc:385] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.863308 12842 raft_consensus.cc:740] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.863744 12840 ts_tablet_manager.cc:1434] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260202 15:07:41.864660 12840 tablet_bootstrap.cc:492] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
W20260202 15:07:41.865211 12615 leader_election.cc:343] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:41.868168 12800 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:41.864135 12842 consensus_queue.cc:260] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.869405 12615 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:41.870527 12840 tablet_bootstrap.cc:654] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.870620 12907 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.870910 12842 ts_tablet_manager.cc:1434] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260202 15:07:41.871265 12907 raft_consensus.cc:697] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:41.872030 12907 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.873391 12842 tablet_bootstrap.cc:492] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.876370 12841 tablet_bootstrap.cc:492] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.876821 12841 ts_tablet_manager.cc:1403] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.008s
I20260202 15:07:41.878553 12842 tablet_bootstrap.cc:654] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.879592 12841 raft_consensus.cc:359] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.880270 12841 raft_consensus.cc:385] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.880527 12841 raft_consensus.cc:740] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.891876 12906 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.900421 12841 consensus_queue.cc:260] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.903894 12841 ts_tablet_manager.cc:1434] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.027s user 0.001s sys 0.005s
I20260202 15:07:41.904790 12841 tablet_bootstrap.cc:492] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.905541 12907 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.906678 12545 catalog_manager.cc:5654] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:41.910831 12841 tablet_bootstrap.cc:654] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.912546 12840 tablet_bootstrap.cc:492] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:41.912921 12840 ts_tablet_manager.cc:1403] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.007s
I20260202 15:07:41.916887 12840 raft_consensus.cc:359] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.917428 12840 raft_consensus.cc:385] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.917651 12840 raft_consensus.cc:740] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.918635 12840 consensus_queue.cc:260] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.920672 12844 consensus_queue.cc:1048] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:07:41.921567 12596 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:07:41.924952 12852 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.930157 12840 ts_tablet_manager.cc:1434] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260202 15:07:41.931190 12840 tablet_bootstrap.cc:492] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:41.935673 12906 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.936900 12842 tablet_bootstrap.cc:492] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:41.937107 12844 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.937474 12842 ts_tablet_manager.cc:1403] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.064s user 0.026s sys 0.006s
I20260202 15:07:41.937647 12844 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.940292 12840 tablet_bootstrap.cc:654] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.940668 12844 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:41.942600 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.940528 12842 raft_consensus.cc:359] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.943449 12842 raft_consensus.cc:385] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.943431 12650 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:41.943913 12842 raft_consensus.cc:740] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.944690 12842 consensus_queue.cc:260] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.945958 12763 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.944693 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:41.947009 12723 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:41.947337 12844 raft_consensus.cc:2804] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:41.947667 12844 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:41.947995 12844 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.948804 12864 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:41.949442 12841 tablet_bootstrap.cc:492] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.949852 12841 ts_tablet_manager.cc:1403] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.015s
I20260202 15:07:41.947019 12842 ts_tablet_manager.cc:1434] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260202 15:07:41.951499 12842 tablet_bootstrap.cc:492] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:41.953346 12841 raft_consensus.cc:359] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.953939 12841 raft_consensus.cc:385] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:41.954152 12841 raft_consensus.cc:740] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:41.954746 12841 consensus_queue.cc:260] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.956324 12844 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.956975 12841 ts_tablet_manager.cc:1434] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260202 15:07:41.957718 12841 tablet_bootstrap.cc:492] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:41.958643 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:41.959179 12649 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.963876 12906 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.964323 12906 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.965862 12649 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:41.966138 12906 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.963459 12841 tablet_bootstrap.cc:654] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:41.966859 12763 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:41.967742 12896 raft_consensus.cc:2804] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:41.968788 12896 raft_consensus.cc:697] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:41.969565 12896 consensus_queue.cc:237] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.975256 12864 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.975761 12649 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.975816 12864 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.978688 12864 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:41.979625 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:41.979990 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:41.980672 12797 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 1: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:07:41.981696 12690 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:41.982493 12864 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:41.988176 12863 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.990489 12863 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:41.991492 12841 tablet_bootstrap.cc:492] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:41.992025 12841 ts_tablet_manager.cc:1403] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.000s
I20260202 15:07:41.992159 12906 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:41.992549 12906 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:41.994002 12906 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.028228 12864 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.029745 12842 tablet_bootstrap.cc:654] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.024627 12844 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:42.032153 12898 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.038828 12898 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.035923 12841 raft_consensus.cc:359] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.037520 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.038249 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.039717 12841 raft_consensus.cc:385] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.040943 12841 raft_consensus.cc:740] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.041910 12841 consensus_queue.cc:260] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.047370 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:42.048085 12724 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.049119 12906 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260202 15:07:42.055444 12841 ts_tablet_manager.cc:1434] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.063s user 0.011s sys 0.011s
I20260202 15:07:42.056488 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.057039 12841 tablet_bootstrap.cc:492] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
W20260202 15:07:42.058838 12615 leader_election.cc:343] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.044198 12800 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:42.061252 12844 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.064355 12844 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.069054 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.071792 12898 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.105141 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.105361 12844 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:42.108016 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.111128 12615 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:42.119697 12841 tablet_bootstrap.cc:654] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.120529 12840 tablet_bootstrap.cc:492] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:42.120903 12840 ts_tablet_manager.cc:1403] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.190s user 0.034s sys 0.046s
I20260202 15:07:42.125277 12840 raft_consensus.cc:359] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.125841 12840 raft_consensus.cc:385] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.126106 12840 raft_consensus.cc:740] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.126225 12980 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:07:42.124115 12615 leader_election.cc:343] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.116981 12898 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:42.126681 12840 consensus_queue.cc:260] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.126749 12980 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.128786 12906 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260202 15:07:42.123118 12725 raft_consensus.cc:2393] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 1: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:07:42.130188 12898 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:42.132779 12898 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.122251 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.135087 12798 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:42.135231 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:07:42.134588 12615 leader_election.cc:343] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.153497 12840 ts_tablet_manager.cc:1434] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.032s user 0.001s sys 0.005s
I20260202 15:07:42.154273 12840 tablet_bootstrap.cc:492] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:42.160511 12844 consensus_queue.cc:1048] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:42.160142 12546 catalog_manager.cc:5654] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:42.162616 12763 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:42.163632 12896 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.168774 12842 tablet_bootstrap.cc:492] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:42.169256 12842 ts_tablet_manager.cc:1403] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.218s user 0.018s sys 0.016s
I20260202 15:07:42.153424 12615 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:42.170290 12906 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.170650 12724 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
W20260202 15:07:42.171401 12615 leader_election.cc:343] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.169570 12898 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.173666 12896 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.174243 12898 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.181721 12980 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:42.192118 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:42.192981 12648 raft_consensus.cc:2393] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 1: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:07:42.195842 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.198000 12765 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:42.199040 12980 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.198926 12615 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:42.205822 12842 raft_consensus.cc:359] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.206744 12842 raft_consensus.cc:385] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.207008 12842 raft_consensus.cc:740] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.207779 12842 consensus_queue.cc:260] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.202992 12906 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:42.209205 12906 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:42.208700 12853 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:42.213657 12842 ts_tablet_manager.cc:1434] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.044s user 0.005s sys 0.000s
I20260202 15:07:42.214038 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:42.214514 12798 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:42.215039 12615 consensus_peers.cc:597] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:42.209581 12906 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.209658 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:42.225750 12842 tablet_bootstrap.cc:492] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:42.233054 12798 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:42.233695 12980 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.233826 12797 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.244136 12840 tablet_bootstrap.cc:654] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.249285 12977 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:42.251053 12852 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:42.255101 12906 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.256259 12852 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:07:42.257215 12615 leader_election.cc:343] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.257110 12898 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:42.257865 12853 consensus_queue.cc:1048] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:42.258234 12841 tablet_bootstrap.cc:492] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:42.258646 12841 ts_tablet_manager.cc:1403] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.202s user 0.016s sys 0.025s
I20260202 15:07:42.260568 12898 consensus_queue.cc:1048] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:42.261776 12841 raft_consensus.cc:359] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.262065 12898 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:42.262343 12841 raft_consensus.cc:385] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.262565 12841 raft_consensus.cc:740] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
W20260202 15:07:42.265813 12982 log.cc:927] Time spent T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260202 15:07:42.266825 12615 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:42.267761 12841 consensus_queue.cc:260] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.268865 12853 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:42.270593 12853 raft_consensus.cc:697] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:42.256289 12906 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.271988 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:42.274216 12725 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.274442 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:42.274914 12798 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.271904 12853 consensus_queue.cc:237] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.280143 12841 ts_tablet_manager.cc:1434] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260202 15:07:42.281280 12841 tablet_bootstrap.cc:492] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
W20260202 15:07:42.297603 12985 log.cc:927] Time spent T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260202 15:07:42.309434 12842 tablet_bootstrap.cc:654] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.326660 12976 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260202 15:07:42.344674 12840 tablet_bootstrap.cc:492] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:42.345183 12840 ts_tablet_manager.cc:1403] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.191s user 0.023s sys 0.034s
I20260202 15:07:42.337014 12798 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:42.351614 12898 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.353724 12840 raft_consensus.cc:359] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.355234 12840 raft_consensus.cc:385] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.355559 12840 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.356269 12840 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.373589 12840 ts_tablet_manager.cc:1434] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
W20260202 15:07:42.376019 12986 log.cc:927] Time spent T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260202 15:07:42.357810 12898 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.387292 12840 tablet_bootstrap.cc:492] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:42.382535 12976 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.383715 12906 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260202 15:07:42.436489 12864 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.437047 12864 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.452383 12898 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.461957 12864 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.464928 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:42.449483 12991 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
W20260202 15:07:42.467523 12690 leader_election.cc:343] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.476085 12841 tablet_bootstrap.cc:654] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.422260 12976 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.482563 12546 catalog_manager.cc:5654] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:42.517189 12725 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
W20260202 15:07:42.497870 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 133519us
I20260202 15:07:42.518623 12844 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:42.519840 12906 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.535789 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.570345 12842 tablet_bootstrap.cc:492] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:42.570811 12842 ts_tablet_manager.cc:1403] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.345s user 0.011s sys 0.033s
I20260202 15:07:42.575119 12842 raft_consensus.cc:359] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.575721 12842 raft_consensus.cc:385] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.575953 12842 raft_consensus.cc:740] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.576455 12842 consensus_queue.cc:260] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.577297 12976 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.578372 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.535737 12906 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.579141 12724 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:42.580526 12906 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.578343 12842 ts_tablet_manager.cc:1434] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260202 15:07:42.581439 12724 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.581605 12842 tablet_bootstrap.cc:492] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:42.563151 12615 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:42.613085 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:42.647213 12615 leader_election.cc:343] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.645306 12844 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260202 15:07:42.660435 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:42.647516 12998 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:42.653429 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:42.663298 12647 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:42.664656 12998 raft_consensus.cc:697] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:42.665537 12998 consensus_queue.cc:237] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.669024 12688 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:42.672343 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.673094 12688 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:42.675065 12864 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.680964 12977 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:42.681339 12977 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:42.681676 12977 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.683530 12841 tablet_bootstrap.cc:492] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:42.683938 12841 ts_tablet_manager.cc:1403] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.403s user 0.019s sys 0.012s
I20260202 15:07:42.684607 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.688063 12977 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.695073 12977 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.703887 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:42.704284 12840 tablet_bootstrap.cc:654] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:42.705384 12690 leader_election.cc:343] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.708808 12841 raft_consensus.cc:359] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.709625 12841 raft_consensus.cc:385] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.709966 12841 raft_consensus.cc:740] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.711671 12615 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:42.712568 12853 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:42.712960 12853 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:42.713310 12853 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:42.720275 12615 leader_election.cc:343] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.721354 12841 consensus_queue.cc:260] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.728137 12997 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.728632 12997 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.731356 12997 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.739117 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:42.740108 12844 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.740932 12844 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.747074 12841 ts_tablet_manager.cc:1434] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.063s user 0.012s sys 0.000s
I20260202 15:07:42.748265 12841 tablet_bootstrap.cc:492] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:42.748618 12906 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:42.752007 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:07:42.755431 12763 leader_election.cc:343] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.751255 12844 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:42.757675 12842 tablet_bootstrap.cc:654] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:42.751978 12615 leader_election.cc:343] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.758873 12840 tablet_bootstrap.cc:492] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:42.759018 12615 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:42.759277 12840 ts_tablet_manager.cc:1403] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.372s user 0.033s sys 0.026s
I20260202 15:07:42.759698 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.759912 12853 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.760299 12907 raft_consensus.cc:2749] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.759999 12546 catalog_manager.cc:5654] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:42.761077 12649 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 1: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:07:42.761667 12853 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.763594 12688 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:42.764320 12977 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.760326 12724 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:42.766033 12841 tablet_bootstrap.cc:654] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.766180 12765 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:42.767727 12980 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:42.768060 12980 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:42.768440 12980 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.771062 12841 tablet_bootstrap.cc:492] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:42.771493 12841 ts_tablet_manager.cc:1403] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.001s
I20260202 15:07:42.773379 12840 raft_consensus.cc:359] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.773974 12840 raft_consensus.cc:385] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.773656 12841 raft_consensus.cc:359] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.774228 12840 raft_consensus.cc:740] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.774384 12841 raft_consensus.cc:385] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.774865 12841 raft_consensus.cc:740] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.774853 12840 consensus_queue.cc:260] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.774974 12980 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.780066 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.780071 12841 consensus_queue.cc:260] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.781368 12840 ts_tablet_manager.cc:1434] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20260202 15:07:42.781649 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:42.782096 12840 tablet_bootstrap.cc:492] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:42.782424 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:42.782598 12841 ts_tablet_manager.cc:1434] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.011s user 0.001s sys 0.008s
I20260202 15:07:42.783542 12841 tablet_bootstrap.cc:492] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:42.784857 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.785811 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:42.786500 12725 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 1: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:07:42.786651 12763 leader_election.cc:343] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:42.787763 12615 leader_election.cc:343] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.788000 12980 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:42.788650 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:42.789225 12723 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.791247 12842 tablet_bootstrap.cc:492] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:42.791636 12842 ts_tablet_manager.cc:1403] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.210s user 0.012s sys 0.008s
I20260202 15:07:42.791705 12841 tablet_bootstrap.cc:654] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:42.792935 12615 leader_election.cc:343] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.793315 12615 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:42.793882 12842 raft_consensus.cc:359] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.794481 12842 raft_consensus.cc:385] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.794678 12905 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.794754 12842 raft_consensus.cc:740] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.795392 12842 consensus_queue.cc:260] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:42.796267 12615 leader_election.cc:343] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.796691 12615 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:42.799079 12853 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.799552 12842 ts_tablet_manager.cc:1434] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260202 15:07:42.800426 12842 tablet_bootstrap.cc:492] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:42.792655 12840 tablet_bootstrap.cc:654] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:42.812902 12615 consensus_peers.cc:597] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:42.814802 12723 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:42.815788 12765 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:42.816480 12844 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:42.816867 12844 raft_consensus.cc:697] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:42.817524 12844 consensus_queue.cc:237] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.820617 12841 tablet_bootstrap.cc:492] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:42.821074 12841 ts_tablet_manager.cc:1403] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.022s
I20260202 15:07:42.823432 12841 raft_consensus.cc:359] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.824003 12841 raft_consensus.cc:385] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.824222 12841 raft_consensus.cc:740] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.824918 12841 consensus_queue.cc:260] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.826756 12841 ts_tablet_manager.cc:1434] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:07:42.827500 12841 tablet_bootstrap.cc:492] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:42.836117 12842 tablet_bootstrap.cc:654] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.852514 12544 catalog_manager.cc:5654] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:42.851315 12907 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260202 15:07:42.855940 12840 tablet_bootstrap.cc:492] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:42.855940 12842 tablet_bootstrap.cc:492] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:42.856412 12842 ts_tablet_manager.cc:1403] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.005s
I20260202 15:07:42.856412 12840 ts_tablet_manager.cc:1403] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.006s
I20260202 15:07:42.859670 12841 tablet_bootstrap.cc:654] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.860128 12840 raft_consensus.cc:359] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.860787 12840 raft_consensus.cc:385] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.861059 12840 raft_consensus.cc:740] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.861646 12840 consensus_queue.cc:260] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.863690 12840 ts_tablet_manager.cc:1434] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260202 15:07:42.864948 12840 tablet_bootstrap.cc:492] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:42.878006 12977 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.878433 12977 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.880187 12977 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.881805 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:42.885093 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.886240 12690 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:42.886196 12842 raft_consensus.cc:359] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.887034 12842 raft_consensus.cc:385] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.887347 12977 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:42.889546 12842 raft_consensus.cc:740] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.890904 12842 consensus_queue.cc:260] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.891633 12840 tablet_bootstrap.cc:654] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.894397 12842 ts_tablet_manager.cc:1434] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.038s user 0.005s sys 0.001s
I20260202 15:07:42.895190 12842 tablet_bootstrap.cc:492] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:42.901888 12841 tablet_bootstrap.cc:492] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:42.902096 12840 tablet_bootstrap.cc:492] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:42.902290 12841 ts_tablet_manager.cc:1403] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.075s user 0.021s sys 0.016s
I20260202 15:07:42.902439 12840 ts_tablet_manager.cc:1403] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.023s
I20260202 15:07:42.904543 12842 tablet_bootstrap.cc:654] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.904671 12841 raft_consensus.cc:359] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.904852 12840 raft_consensus.cc:359] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.905282 12841 raft_consensus.cc:385] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.905397 12840 raft_consensus.cc:385] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.905551 12841 raft_consensus.cc:740] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.905627 12840 raft_consensus.cc:740] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.906105 12840 consensus_queue.cc:260] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.906105 12841 consensus_queue.cc:260] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.907629 12841 ts_tablet_manager.cc:1434] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260202 15:07:42.908703 12841 tablet_bootstrap.cc:492] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:42.910259 12977 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.910386 12840 ts_tablet_manager.cc:1434] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260202 15:07:42.910758 12977 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.911350 12840 tablet_bootstrap.cc:492] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:42.912483 12977 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.913697 12841 tablet_bootstrap.cc:654] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.917054 12840 tablet_bootstrap.cc:654] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.918823 12842 tablet_bootstrap.cc:492] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:42.919239 12842 ts_tablet_manager.cc:1403] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.004s
I20260202 15:07:42.921648 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:42.922196 12649 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:42.923275 12688 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:42.924165 12977 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:42.924464 12977 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:42.924731 12977 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.925755 12842 raft_consensus.cc:359] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.926414 12842 raft_consensus.cc:385] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.926674 12842 raft_consensus.cc:740] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.927297 12842 consensus_queue.cc:260] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.929468 12842 ts_tablet_manager.cc:1434] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.010s user 0.001s sys 0.003s
I20260202 15:07:42.930306 12842 tablet_bootstrap.cc:492] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:42.935065 12842 tablet_bootstrap.cc:654] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.938760 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:42.940843 12690 leader_election.cc:343] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.941803 12841 tablet_bootstrap.cc:492] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:42.942209 12841 ts_tablet_manager.cc:1403] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.016s
I20260202 15:07:42.944685 12841 raft_consensus.cc:359] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.945288 12841 raft_consensus.cc:385] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.945536 12841 raft_consensus.cc:740] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.946055 12841 consensus_queue.cc:260] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.947078 12840 tablet_bootstrap.cc:492] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:42.947443 12841 ts_tablet_manager.cc:1434] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260202 15:07:42.947984 12840 ts_tablet_manager.cc:1403] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.002s
I20260202 15:07:42.949522 12840 raft_consensus.cc:359] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.949936 12840 raft_consensus.cc:385] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.950086 12840 raft_consensus.cc:740] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.950608 12840 consensus_queue.cc:260] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.951834 12841 tablet_bootstrap.cc:492] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:42.952574 12840 ts_tablet_manager.cc:1434] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260202 15:07:42.956670 12841 tablet_bootstrap.cc:654] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.956780 12977 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.965330 12840 tablet_bootstrap.cc:492] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
W20260202 15:07:42.970199 12596 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:07:42.976241 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:42.976714 12649 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:42.977012 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:07:42.978487 12690 leader_election.cc:343] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.979940 12999 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.980851 12977 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.980382 12999 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.984015 12842 tablet_bootstrap.cc:492] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:42.984396 12842 ts_tablet_manager.cc:1403] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.054s user 0.015s sys 0.015s
I20260202 15:07:42.985440 12999 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.985496 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:07:42.986650 12615 leader_election.cc:343] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.986783 12840 tablet_bootstrap.cc:654] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:42.986562 12842 raft_consensus.cc:359] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.986819 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.987171 12842 raft_consensus.cc:385] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:42.987221 12999 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.987505 12842 raft_consensus.cc:740] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.987677 12999 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.987892 12864 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.987844 12977 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:42.988178 12842 consensus_queue.cc:260] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.988368 12864 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.988459 12977 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:42.990079 12864 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.990332 12977 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:42.991844 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:42.993165 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.993296 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:42.993320 12647 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:42.993691 12800 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:42.992509 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:42.994524 12648 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:42.992666 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:42.996551 12688 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:42.997639 12688 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:42.998608 12976 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:42.999050 12976 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:42.999413 12976 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:42.999841 12615 leader_election.cc:343] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:43.001344 12690 leader_election.cc:343] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:42.987453 12799 raft_consensus.cc:2468] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:43.003671 12615 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:43.004307 12997 raft_consensus.cc:2804] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:43.004640 12997 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:43.004914 12997 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.005383 12999 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.006268 12915 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:43.006577 12915 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:43.006855 12915 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.007349 12841 tablet_bootstrap.cc:492] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:43.007720 12841 ts_tablet_manager.cc:1403] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.013s
I20260202 15:07:43.008790 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.009840 12841 raft_consensus.cc:359] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.010502 12841 raft_consensus.cc:385] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260202 15:07:43.010325 12615 leader_election.cc:343] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.010856 12841 raft_consensus.cc:740] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.011025 12615 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:43.011487 12841 consensus_queue.cc:260] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.012107 12999 raft_consensus.cc:2749] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.013322 12841 ts_tablet_manager.cc:1434] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:07:43.014118 12841 tablet_bootstrap.cc:492] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:43.018087 12840 tablet_bootstrap.cc:492] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.018468 12840 ts_tablet_manager.cc:1403] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.023s
I20260202 15:07:43.022255 12976 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.023365 12915 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.024443 12841 tablet_bootstrap.cc:654] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.025168 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:43.024595 12840 raft_consensus.cc:359] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.022445 12997 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.025593 12840 raft_consensus.cc:385] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.025985 12840 raft_consensus.cc:740] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:42.990053 12842 ts_tablet_manager.cc:1434] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:07:43.026131 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:43.026683 12647 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.027325 12649 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:43.028702 12688 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:43.030303 12842 tablet_bootstrap.cc:492] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:43.030125 12977 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:43.031697 12977 raft_consensus.cc:697] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:43.032258 12840 consensus_queue.cc:260] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.033452 12915 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.034073 12840 ts_tablet_manager.cc:1434] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260202 15:07:43.032662 12977 consensus_queue.cc:237] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.034848 12840 tablet_bootstrap.cc:492] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:43.035063 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:43.035494 12796 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.038342 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:07:43.039549 12615 leader_election.cc:343] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.039892 12840 tablet_bootstrap.cc:654] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.041188 12997 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.041676 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:43.042138 12799 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.049969 12842 tablet_bootstrap.cc:654] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.025629 12648 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.055065 12840 tablet_bootstrap.cc:492] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.055588 12840 ts_tablet_manager.cc:1403] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20260202 15:07:43.057621 12840 raft_consensus.cc:359] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.058173 12840 raft_consensus.cc:385] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.058398 12840 raft_consensus.cc:740] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.059036 12796 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:43.058928 12840 consensus_queue.cc:260] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.060074 12690 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:43.062975 12647 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:43.065172 12864 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:43.067634 12864 raft_consensus.cc:697] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:43.068670 12864 consensus_queue.cc:237] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.066190 12648 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:43.066685 12840 ts_tablet_manager.cc:1434] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260202 15:07:43.065868 12545 catalog_manager.cc:5654] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:43.073421 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:43.074574 12840 tablet_bootstrap.cc:492] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
W20260202 15:07:43.075066 12690 leader_election.cc:343] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.072711 12976 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.077615 12688 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:43.093932 12976 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:43.099712 12799 raft_consensus.cc:2468] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:43.100309 12976 raft_consensus.cc:697] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:43.102355 12997 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.103031 12841 tablet_bootstrap.cc:492] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:43.102759 12997 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.103471 12841 ts_tablet_manager.cc:1403] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.090s user 0.036s sys 0.019s
I20260202 15:07:43.102982 12976 consensus_queue.cc:237] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.104744 12997 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.105899 12841 raft_consensus.cc:359] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.106328 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.106328 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:43.106441 12841 raft_consensus.cc:385] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.107460 12841 raft_consensus.cc:740] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.108117 12615 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:43.107988 12841 consensus_queue.cc:260] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.107293 12545 catalog_manager.cc:5654] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
W20260202 15:07:43.111157 12615 leader_election.cc:343] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:43.111685 12615 leader_election.cc:343] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.111876 12841 ts_tablet_manager.cc:1434] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20260202 15:07:43.112025 12615 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:43.112775 12997 raft_consensus.cc:2804] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:43.112880 12898 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.112982 12841 tablet_bootstrap.cc:492] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:43.113214 12997 raft_consensus.cc:697] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:43.113930 12997 consensus_queue.cc:237] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.129000 12840 tablet_bootstrap.cc:654] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.134346 12980 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.140100 12980 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.142932 12980 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:43.143570 12842 tablet_bootstrap.cc:492] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:43.143985 12842 ts_tablet_manager.cc:1403] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.114s user 0.016s sys 0.007s
I20260202 15:07:43.146013 12842 raft_consensus.cc:359] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.146549 12842 raft_consensus.cc:385] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.146313 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:43.146785 12842 raft_consensus.cc:740] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.147261 12724 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 1: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:07:43.147289 12842 consensus_queue.cc:260] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.147949 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.148563 12647 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 1: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:07:43.149279 12842 ts_tablet_manager.cc:1434] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260202 15:07:43.149982 12842 tablet_bootstrap.cc:492] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:43.151278 12977 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.151654 12977 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.153314 12977 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.154259 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.156141 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.157346 12841 tablet_bootstrap.cc:654] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.157191 12688 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:43.158181 12977 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.160355 12765 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:43.161393 12865 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.163077 12547 catalog_manager.cc:5654] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:43.167362 12865 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.172984 12840 tablet_bootstrap.cc:492] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.173381 12840 ts_tablet_manager.cc:1403] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.099s user 0.014s sys 0.005s
I20260202 15:07:43.175602 12840 raft_consensus.cc:359] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.176168 12840 raft_consensus.cc:385] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.176395 12840 raft_consensus.cc:740] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.176879 12840 consensus_queue.cc:260] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.176702 12545 catalog_manager.cc:5654] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:43.179261 12840 ts_tablet_manager.cc:1434] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260202 15:07:43.180055 12840 tablet_bootstrap.cc:492] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:43.181968 12841 tablet_bootstrap.cc:492] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:43.182408 12841 ts_tablet_manager.cc:1403] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.005s
I20260202 15:07:43.184762 12841 raft_consensus.cc:359] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.185374 12841 raft_consensus.cc:385] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.185607 12840 tablet_bootstrap.cc:654] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.185649 12841 raft_consensus.cc:740] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.186520 12841 consensus_queue.cc:260] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.189522 12842 tablet_bootstrap.cc:654] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.190532 12841 ts_tablet_manager.cc:1434] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260202 15:07:43.191282 12841 tablet_bootstrap.cc:492] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:43.194239 12842 tablet_bootstrap.cc:492] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:43.194583 12842 ts_tablet_manager.cc:1403] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20260202 15:07:43.194492 12724 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.196980 12841 tablet_bootstrap.cc:654] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.196877 12842 raft_consensus.cc:359] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.197419 12842 raft_consensus.cc:385] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.197636 12842 raft_consensus.cc:740] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.198199 12842 consensus_queue.cc:260] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.201680 12853 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.202278 12853 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.205072 12840 tablet_bootstrap.cc:492] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.205453 12840 ts_tablet_manager.cc:1403] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20260202 15:07:43.206990 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.207535 12840 raft_consensus.cc:359] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.208092 12840 raft_consensus.cc:385] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.208335 12840 raft_consensus.cc:740] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.208921 12840 consensus_queue.cc:260] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.209353 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:43.210466 12723 raft_consensus.cc:2393] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 1: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:07:43.220846 12840 ts_tablet_manager.cc:1434] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260202 15:07:43.215482 12997 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.221676 12840 tablet_bootstrap.cc:492] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:43.221899 12997 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.224485 12997 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.225184 12853 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.227332 12615 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:43.230370 12842 ts_tablet_manager.cc:1434] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.035s user 0.008s sys 0.000s
I20260202 15:07:43.228560 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:43.228983 12840 tablet_bootstrap.cc:654] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.229172 12865 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:43.233647 12898 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.229689 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.241622 12796 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:43.243521 12842 tablet_bootstrap.cc:492] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:43.243942 12615 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:43.245143 12840 tablet_bootstrap.cc:492] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.245888 12723 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.246892 12999 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.247490 12999 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.249922 12999 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.250469 12999 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:43.250958 12999 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:43.251372 12999 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.252290 12840 ts_tablet_manager.cc:1403] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.012s
I20260202 15:07:43.253518 12842 tablet_bootstrap.cc:654] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.254077 12841 tablet_bootstrap.cc:492] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:43.254442 12841 ts_tablet_manager.cc:1403] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.063s user 0.031s sys 0.024s
I20260202 15:07:43.254717 12840 raft_consensus.cc:359] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.255329 12840 raft_consensus.cc:385] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.255548 12840 raft_consensus.cc:740] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
W20260202 15:07:43.255986 12615 leader_election.cc:343] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.256058 12840 consensus_queue.cc:260] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.257045 12841 raft_consensus.cc:359] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.257613 12841 raft_consensus.cc:385] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.257886 12841 raft_consensus.cc:740] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.258550 12841 consensus_queue.cc:260] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.260541 12841 ts_tablet_manager.cc:1434] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:07:43.261512 12841 tablet_bootstrap.cc:492] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:43.262262 12840 ts_tablet_manager.cc:1434] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260202 15:07:43.263285 12840 tablet_bootstrap.cc:492] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:43.264068 12976 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.264518 12976 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.266198 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:43.268006 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.268075 12976 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.268597 12647 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:43.270268 12688 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:43.270491 12841 tablet_bootstrap.cc:654] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.271716 12976 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:43.272053 12976 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:43.272372 12976 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.273021 12842 tablet_bootstrap.cc:492] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:43.273481 12842 ts_tablet_manager.cc:1403] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.015s
I20260202 15:07:43.276110 12842 raft_consensus.cc:359] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.276693 12842 raft_consensus.cc:385] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.278283 12840 tablet_bootstrap.cc:654] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.278964 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.279690 12796 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:43.286185 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.276961 12842 raft_consensus.cc:740] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.287737 12842 consensus_queue.cc:260] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.288713 12840 tablet_bootstrap.cc:492] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.289074 12840 ts_tablet_manager.cc:1403] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.008s
I20260202 15:07:43.291668 12840 raft_consensus.cc:359] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.292181 12840 raft_consensus.cc:385] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.292409 12840 raft_consensus.cc:740] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.294377 12898 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.298298 12841 tablet_bootstrap.cc:492] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:43.298725 12842 ts_tablet_manager.cc:1434] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260202 15:07:43.308281 12841 ts_tablet_manager.cc:1403] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.025s
I20260202 15:07:43.309134 12842 tablet_bootstrap.cc:492] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:43.310642 12841 raft_consensus.cc:359] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.311360 12841 raft_consensus.cc:385] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.311594 12841 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.312104 12841 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.313580 12841 ts_tablet_manager.cc:1434] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260202 15:07:43.315294 12841 tablet_bootstrap.cc:492] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:43.298591 12840 consensus_queue.cc:260] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.319005 12840 ts_tablet_manager.cc:1434] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.030s user 0.003s sys 0.004s
I20260202 15:07:43.319733 12840 tablet_bootstrap.cc:492] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:43.320353 12841 tablet_bootstrap.cc:654] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.325048 12840 tablet_bootstrap.cc:654] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.325929 12980 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:43.326941 12842 tablet_bootstrap.cc:654] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.300274 12999 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.330397 12999 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:07:43.301561 12615 leader_election.cc:343] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:43.331851 12615 leader_election.cc:343] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.332394 12615 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:43.333132 12840 tablet_bootstrap.cc:492] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.333456 12840 ts_tablet_manager.cc:1403] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20260202 15:07:43.334331 12905 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.304337 12976 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.334923 12905 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:43.335309 12840 raft_consensus.cc:359] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.335831 12840 raft_consensus.cc:385] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.336045 12840 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.331229 12842 tablet_bootstrap.cc:492] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:43.336362 12976 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.336951 12842 ts_tablet_manager.cc:1403] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.002s
I20260202 15:07:43.336630 12840 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.338500 12840 ts_tablet_manager.cc:1434] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:07:43.338354 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:43.339213 12840 tablet_bootstrap.cc:492] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:43.339092 12796 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.339051 12842 raft_consensus.cc:359] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.340541 12842 raft_consensus.cc:385] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.340799 12842 raft_consensus.cc:740] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.341554 12842 consensus_queue.cc:260] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.341951 12977 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.350241 12842 ts_tablet_manager.cc:1434] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.013s user 0.002s sys 0.009s
I20260202 15:07:43.349958 12977 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.351156 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:43.351857 12647 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.353269 12840 tablet_bootstrap.cc:654] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.353307 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:43.354526 12690 leader_election.cc:343] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.355068 12977 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.356159 12842 tablet_bootstrap.cc:492] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:43.360030 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.360646 12650 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:43.362248 12688 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:43.364202 12842 tablet_bootstrap.cc:654] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.369124 12840 tablet_bootstrap.cc:492] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.369510 12840 ts_tablet_manager.cc:1403] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.007s
I20260202 15:07:43.371279 12976 raft_consensus.cc:2804] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:43.371714 12976 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:43.372085 12976 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.372143 12840 raft_consensus.cc:359] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.372742 12840 raft_consensus.cc:385] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.372957 12840 raft_consensus.cc:740] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.373699 12840 consensus_queue.cc:260] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.376508 12842 tablet_bootstrap.cc:492] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:43.376896 12842 ts_tablet_manager.cc:1403] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.003s
I20260202 15:07:43.379119 12840 ts_tablet_manager.cc:1434] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260202 15:07:43.378942 12842 raft_consensus.cc:359] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.379611 12842 raft_consensus.cc:385] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.379853 12840 tablet_bootstrap.cc:492] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:43.379875 12842 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.380788 12842 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.380963 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:43.382983 12842 ts_tablet_manager.cc:1434] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:07:43.383886 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:43.384421 12799 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.386957 12840 tablet_bootstrap.cc:654] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.391332 12799 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:43.383865 12842 tablet_bootstrap.cc:492] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:43.393961 12898 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:43.398479 12842 tablet_bootstrap.cc:654] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.400128 12615 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:43.408128 12840 tablet_bootstrap.cc:492] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.408593 12840 ts_tablet_manager.cc:1403] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.008s
I20260202 15:07:43.411552 12840 raft_consensus.cc:359] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.415134 12840 raft_consensus.cc:385] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.415429 12840 raft_consensus.cc:740] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.416307 12840 consensus_queue.cc:260] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.419291 12840 ts_tablet_manager.cc:1434] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260202 15:07:43.420279 12840 tablet_bootstrap.cc:492] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:43.425737 12842 tablet_bootstrap.cc:492] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:43.427769 12840 tablet_bootstrap.cc:654] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.426115 12647 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:43.504344 12688 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:43.505401 12864 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:43.505977 12864 raft_consensus.cc:697] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:43.506623 12844 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.506970 12864 consensus_queue.cc:237] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.507011 12844 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.512171 12844 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:43.426151 12842 ts_tablet_manager.cc:1403] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.042s user 0.021s sys 0.011s
I20260202 15:07:43.513047 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:43.514590 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.515524 12650 raft_consensus.cc:2393] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 1: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:07:43.516752 12852 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.514806 12842 raft_consensus.cc:359] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.517009 12763 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:43.517195 12852 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.517479 12842 raft_consensus.cc:385] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.518328 12842 raft_consensus.cc:740] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.518828 12852 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.519122 12844 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.518997 12842 consensus_queue.cc:260] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.520288 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.520874 12799 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:43.521507 12842 ts_tablet_manager.cc:1434] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260202 15:07:43.521838 12690 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:43.522382 12852 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:43.522675 12852 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:43.522825 12840 tablet_bootstrap.cc:492] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:43.523029 12852 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.523327 12840 ts_tablet_manager.cc:1403] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.103s user 0.033s sys 0.045s
I20260202 15:07:43.525513 12840 raft_consensus.cc:359] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.526167 12840 raft_consensus.cc:385] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.526429 12840 raft_consensus.cc:740] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.527350 12840 consensus_queue.cc:260] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.532974 12852 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.535663 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:43.536257 12649 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.537117 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.491212 12977 consensus_queue.cc:1048] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
I20260202 15:07:43.541062 12915 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.541585 12915 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.544484 12915 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:07:43.467419 12615 leader_election.cc:343] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:43.549791 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 82411.7us
I20260202 15:07:43.553542 12649 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:43.554351 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.554435 12688 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:43.555029 12800 raft_consensus.cc:2393] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 1: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:07:43.553418 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.556097 12690 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:43.556761 12864 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:43.557021 12915 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.557299 12864 raft_consensus.cc:697] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:43.558214 12864 consensus_queue.cc:237] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.562832 12852 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.564293 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:43.564745 12800 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.565109 12915 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.565662 13061 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:43.426759 12980 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.566854 12980 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.570171 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:07:43.571549 12765 leader_election.cc:343] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.571787 12800 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:43.572188 12980 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:43.434592 12853 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:43.573328 12853 raft_consensus.cc:697] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:43.574597 12800 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.461264 12796 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:43.577329 12977 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:43.499823 12976 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.586496 12842 tablet_bootstrap.cc:492] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:43.588652 12898 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.589094 12898 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.591753 12898 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.592857 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:43.593462 12645 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.596370 13058 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:43.599560 12645 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:43.600641 12688 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:43.600909 12844 raft_consensus.cc:2749] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.601481 13058 raft_consensus.cc:2804] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:43.601989 13058 raft_consensus.cc:697] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:43.611673 12840 ts_tablet_manager.cc:1434] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.088s user 0.024s sys 0.000s
I20260202 15:07:43.620223 12840 tablet_bootstrap.cc:492] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:43.623378 12976 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.574255 12853 consensus_queue.cc:237] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.623107 13058 consensus_queue.cc:237] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.638132 12977 consensus_queue.cc:1048] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260202 15:07:43.643301 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:43.643846 12724 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:43.642359 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:43.650177 12976 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.650686 12976 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.652856 12976 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.653589 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:43.645646 12690 leader_election.cc:343] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.656335 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.666843 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.667389 12649 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:43.671845 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:07:43.666215 12690 leader_election.cc:343] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:43.685009 12688 leader_election.cc:343] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:43.712013 12615 leader_election.cc:343] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:43.761584 13066 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260202 15:07:43.764142 12840 tablet_bootstrap.cc:654] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:43.730005 13070 log.cc:927] Time spent T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260202 15:07:43.774228 13059 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.774808 13059 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:43.767318 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 77652.3us
I20260202 15:07:43.813416 13058 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.814399 13058 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.822345 13058 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.828569 12841 tablet_bootstrap.cc:492] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:43.829088 12841 ts_tablet_manager.cc:1403] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.514s user 0.184s sys 0.257s
I20260202 15:07:43.739289 12763 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:43.834583 13059 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:07:43.840919 13067 log.cc:927] Time spent T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.176s user 0.003s sys 0.000s
I20260202 15:07:43.844920 13057 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:43.846920 13057 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:43.847203 13057 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.848261 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.850793 12615 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
W20260202 15:07:43.852814 12690 leader_election.cc:343] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.852737 13069 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:43.853561 13069 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:43.852313 12864 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:43.853858 13069 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.854580 12842 tablet_bootstrap.cc:654] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.854532 12688 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:07:43.855402 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 182188us
I20260202 15:07:43.857144 12915 raft_consensus.cc:2749] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260202 15:07:43.847888 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 118092us
I20260202 15:07:43.849974 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:43.859149 12841 raft_consensus.cc:359] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.860734 12841 raft_consensus.cc:385] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.861042 12841 raft_consensus.cc:740] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.861629 12841 consensus_queue.cc:260] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.867117 12905 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:07:43.871420 12690 leader_election.cc:343] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.873514 12841 ts_tablet_manager.cc:1434] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20260202 15:07:43.874292 12841 tablet_bootstrap.cc:492] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:43.880704 12980 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:43.880548 12544 catalog_manager.cc:5654] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:43.884479 12544 catalog_manager.cc:5654] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:43.885447 12544 catalog_manager.cc:5654] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:43.887208 12980 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.888893 12980 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:43.890245 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:43.890892 12721 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:43.892009 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.892035 12765 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:43.893568 12844 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:43.893989 12844 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:43.894336 12844 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.894294 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:07:43.900676 12688 leader_election.cc:343] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.901187 12688 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:43.902158 12915 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.903808 13057 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.904050 12844 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:43.907429 12688 leader_election.cc:343] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.902117 12547 catalog_manager.cc:5654] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
W20260202 15:07:43.882383 13064 log.cc:927] Time spent T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
I20260202 15:07:43.915401 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:43.917250 13057 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:43.926121 12688 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:07:43.927520 12765 leader_election.cc:343] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.928226 12915 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:43.930605 13069 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.944985 13069 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:43.954854 12841 tablet_bootstrap.cc:654] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:43.954968 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:43.955637 12646 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:43.958166 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:43.958678 12648 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.960494 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:43.961012 12724 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.962014 12853 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:43.967319 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:43.973848 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:43.974344 12722 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:43.974745 12844 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:07:43.975585 12615 leader_election.cc:343] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:43.979287 12724 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:43.980559 12615 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:43.982159 13068 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:43.983388 12842 tablet_bootstrap.cc:492] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:43.983726 12842 ts_tablet_manager.cc:1403] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.397s user 0.023s sys 0.011s
I20260202 15:07:43.982561 13068 raft_consensus.cc:697] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:43.985227 12648 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:43.985110 13068 consensus_queue.cc:237] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.984369 12915 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260202 15:07:43.989624 12842 raft_consensus.cc:359] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:43.991017 12842 raft_consensus.cc:385] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:43.991292 12842 raft_consensus.cc:740] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:43.996635 12915 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:44.017241 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:44.017876 12648 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.021972 12763 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:44.022898 13057 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:44.023295 13057 raft_consensus.cc:697] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:44.024005 13057 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.053380 12915 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.059389 12840 tablet_bootstrap.cc:492] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:44.059828 12840 ts_tablet_manager.cc:1403] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.440s user 0.022s sys 0.029s
I20260202 15:07:44.059860 12842 consensus_queue.cc:260] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.069089 12840 raft_consensus.cc:359] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.070024 12844 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.070662 12840 raft_consensus.cc:385] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.071470 12840 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.072160 12842 ts_tablet_manager.cc:1434] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.088s user 0.009s sys 0.017s
I20260202 15:07:44.071910 12844 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.075111 12842 tablet_bootstrap.cc:492] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:44.075578 12844 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:44.077919 12840 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.077593 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.080938 12841 tablet_bootstrap.cc:492] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.081358 12841 ts_tablet_manager.cc:1403] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.207s user 0.031s sys 0.012s
I20260202 15:07:44.085368 12840 ts_tablet_manager.cc:1434] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20260202 15:07:44.086127 12840 tablet_bootstrap.cc:492] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:44.122485 12722 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:44.132520 12841 raft_consensus.cc:359] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.133141 12841 raft_consensus.cc:385] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.132679 12765 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:44.133383 12841 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.133760 12844 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:44.134158 12844 raft_consensus.cc:697] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:44.134040 12841 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.124343 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:44.134797 12844 consensus_queue.cc:237] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.137531 12907 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.140568 12841 ts_tablet_manager.cc:1434] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.059s user 0.025s sys 0.011s
I20260202 15:07:44.140584 12840 tablet_bootstrap.cc:654] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.151825 12841 tablet_bootstrap.cc:492] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.160343 13058 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.156137 12648 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:44.153131 12915 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260202 15:07:44.153036 12907 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.165709 13058 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.167866 12907 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.167098 13058 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.171854 12763 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:44.173431 12844 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:44.177888 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:44.183461 13058 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.189349 12907 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.213675 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.214242 12722 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:44.217681 12545 catalog_manager.cc:5654] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:44.224820 12545 catalog_manager.cc:5654] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
W20260202 15:07:44.220011 12596 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:07:44.220939 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:07:44.228227 12615 leader_election.cc:343] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.230388 12615 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:44.233305 12841 tablet_bootstrap.cc:654] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.233610 12898 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:44.234474 12842 tablet_bootstrap.cc:654] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.236646 12898 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:44.236994 12898 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.235152 12800 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:44.236375 12615 consensus_peers.cc:597] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:44.261471 12898 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.263253 12898 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.263706 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:44.264209 12724 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.265844 12864 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.280313 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:44.280478 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:44.284188 12690 leader_election.cc:343] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:44.284801 12615 leader_election.cc:343] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:44.286670 12688 leader_election.cc:343] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.312275 12840 tablet_bootstrap.cc:492] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:44.312671 12840 ts_tablet_manager.cc:1403] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.227s user 0.028s sys 0.004s
I20260202 15:07:44.312997 12688 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:44.316370 12915 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:44.333056 12724 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:44.333984 12615 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:44.335085 12907 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:44.335578 12841 tablet_bootstrap.cc:492] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.335549 12907 raft_consensus.cc:697] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:44.336241 12841 ts_tablet_manager.cc:1403] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.185s user 0.015s sys 0.012s
I20260202 15:07:44.338011 12842 tablet_bootstrap.cc:492] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:44.338232 12907 consensus_queue.cc:237] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.346153 12842 ts_tablet_manager.cc:1403] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.271s user 0.018s sys 0.029s
I20260202 15:07:44.346344 12841 raft_consensus.cc:359] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.346886 12841 raft_consensus.cc:385] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.347147 12841 raft_consensus.cc:740] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.347862 12841 consensus_queue.cc:260] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.349646 12841 ts_tablet_manager.cc:1434] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260202 15:07:44.350666 12841 tablet_bootstrap.cc:492] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.353168 12840 raft_consensus.cc:359] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.353763 12840 raft_consensus.cc:385] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.354008 12840 raft_consensus.cc:740] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.355219 12999 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.355603 12999 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.356223 12841 tablet_bootstrap.cc:654] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.356362 12842 raft_consensus.cc:359] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.356963 12842 raft_consensus.cc:385] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.357165 12999 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.357218 12842 raft_consensus.cc:740] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.357650 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.358163 12724 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:44.358386 12840 consensus_queue.cc:260] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.359078 12615 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:44.359772 12999 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:44.360152 12999 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:44.360527 12999 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.357853 12842 consensus_queue.cc:260] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.364912 12840 ts_tablet_manager.cc:1434] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.052s user 0.005s sys 0.001s
I20260202 15:07:44.366493 12840 tablet_bootstrap.cc:492] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:44.369292 12547 catalog_manager.cc:5654] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:44.372315 12999 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.373061 12840 tablet_bootstrap.cc:654] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.358726 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:44.376251 12842 ts_tablet_manager.cc:1434] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20260202 15:07:44.377233 12841 tablet_bootstrap.cc:492] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.378633 12841 ts_tablet_manager.cc:1403] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.016s
I20260202 15:07:44.378988 12842 tablet_bootstrap.cc:492] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:44.380723 12841 raft_consensus.cc:359] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.381377 12841 raft_consensus.cc:385] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.381611 12841 raft_consensus.cc:740] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.378584 12799 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:44.382109 12841 consensus_queue.cc:260] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.384042 12841 ts_tablet_manager.cc:1434] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:07:44.384429 12842 tablet_bootstrap.cc:654] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.384765 12841 tablet_bootstrap.cc:492] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.385229 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:44.385715 12724 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.391059 12841 tablet_bootstrap.cc:654] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.391744 12724 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:44.392590 12615 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:44.393226 12853 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:44.393594 12853 raft_consensus.cc:697] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:44.394251 12853 consensus_queue.cc:237] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.399809 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:44.400395 12799 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.401827 12841 tablet_bootstrap.cc:492] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.402086 12999 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.402184 12841 ts_tablet_manager.cc:1403] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20260202 15:07:44.404608 12841 raft_consensus.cc:359] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.405196 12841 raft_consensus.cc:385] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.405431 12841 raft_consensus.cc:740] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.406613 12842 tablet_bootstrap.cc:492] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:44.406972 12842 ts_tablet_manager.cc:1403] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.003s
I20260202 15:07:44.407526 12799 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:44.405902 12841 consensus_queue.cc:260] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.408828 12842 raft_consensus.cc:359] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.409353 12842 raft_consensus.cc:385] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.409581 12842 raft_consensus.cc:740] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.410389 12841 ts_tablet_manager.cc:1434] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260202 15:07:44.410323 12842 consensus_queue.cc:260] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.413465 12842 ts_tablet_manager.cc:1434] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260202 15:07:44.414166 12842 tablet_bootstrap.cc:492] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:44.414041 12853 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.411873 12841 tablet_bootstrap.cc:492] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.421916 12841 tablet_bootstrap.cc:654] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.426712 12842 tablet_bootstrap.cc:654] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.434669 12840 tablet_bootstrap.cc:492] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:44.435182 12840 ts_tablet_manager.cc:1403] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.069s user 0.015s sys 0.014s
I20260202 15:07:44.435247 12547 catalog_manager.cc:5654] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:44.438402 12840 raft_consensus.cc:359] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.439153 12840 raft_consensus.cc:385] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.439376 12840 raft_consensus.cc:740] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.439975 12840 consensus_queue.cc:260] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.442720 12840 ts_tablet_manager.cc:1434] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260202 15:07:44.443653 12799 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.446063 12842 tablet_bootstrap.cc:492] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:44.446595 12842 ts_tablet_manager.cc:1403] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20260202 15:07:44.436725 12547 catalog_manager.cc:5654] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:44.449381 12842 raft_consensus.cc:359] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.450806 12842 raft_consensus.cc:385] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.451197 12842 raft_consensus.cc:740] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.452188 12840 tablet_bootstrap.cc:492] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:44.453339 12915 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.453802 12915 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.454030 12999 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.455420 12915 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.456826 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:44.457044 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:44.458246 12688 leader_election.cc:343] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:44.458861 12690 leader_election.cc:343] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.459323 12690 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:44.460511 12915 raft_consensus.cc:2749] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:44.460625 12999 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.461544 12842 consensus_queue.cc:260] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.461249 12999 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.465134 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:07:44.466136 12615 leader_election.cc:343] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.467167 12999 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.467684 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:44.469087 12615 leader_election.cc:343] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.469530 12615 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:44.470254 12999 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:44.471306 12842 ts_tablet_manager.cc:1434] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260202 15:07:44.472281 12842 tablet_bootstrap.cc:492] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:44.481252 12840 tablet_bootstrap.cc:654] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.486203 12842 tablet_bootstrap.cc:654] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.490036 12842 tablet_bootstrap.cc:492] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:44.490425 12842 ts_tablet_manager.cc:1403] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.002s
I20260202 15:07:44.492410 12841 tablet_bootstrap.cc:492] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.492784 12841 ts_tablet_manager.cc:1403] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.081s user 0.018s sys 0.045s
I20260202 15:07:44.493620 12840 tablet_bootstrap.cc:492] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:44.493964 12840 ts_tablet_manager.cc:1403] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.000s
I20260202 15:07:44.494835 12841 raft_consensus.cc:359] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.495383 12841 raft_consensus.cc:385] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.495594 12841 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.495980 12840 raft_consensus.cc:359] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.496107 12841 consensus_queue.cc:260] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.496531 12840 raft_consensus.cc:385] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.496799 12840 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.496269 12842 raft_consensus.cc:359] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.497344 12842 raft_consensus.cc:385] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.497617 12842 raft_consensus.cc:740] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.497507 12840 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.498327 12841 ts_tablet_manager.cc:1434] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260202 15:07:44.498263 12842 consensus_queue.cc:260] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.499298 12841 tablet_bootstrap.cc:492] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.499639 12840 ts_tablet_manager.cc:1434] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260202 15:07:44.500546 12842 ts_tablet_manager.cc:1434] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260202 15:07:44.500974 12840 tablet_bootstrap.cc:492] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:44.501395 12842 tablet_bootstrap.cc:492] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:44.504694 12841 tablet_bootstrap.cc:654] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.506116 12840 tablet_bootstrap.cc:654] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.506775 12842 tablet_bootstrap.cc:654] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.516999 12840 tablet_bootstrap.cc:492] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:44.517344 12840 ts_tablet_manager.cc:1403] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260202 15:07:44.519428 12840 raft_consensus.cc:359] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.520089 12840 raft_consensus.cc:385] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.531503 12841 tablet_bootstrap.cc:492] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.520644 12840 raft_consensus.cc:740] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.532032 12841 ts_tablet_manager.cc:1403] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.007s
I20260202 15:07:44.532243 12840 consensus_queue.cc:260] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.534278 12840 ts_tablet_manager.cc:1434] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260202 15:07:44.533921 13057 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.534027 12841 raft_consensus.cc:359] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.534662 12841 raft_consensus.cc:385] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.534950 12841 raft_consensus.cc:740] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.535568 12841 consensus_queue.cc:260] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.537458 12841 ts_tablet_manager.cc:1434] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:07:44.538573 12841 tablet_bootstrap.cc:492] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.545080 12840 tablet_bootstrap.cc:492] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:44.547307 12841 tablet_bootstrap.cc:654] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.551944 12840 tablet_bootstrap.cc:654] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.555927 12865 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.556006 12724 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.561450 12841 tablet_bootstrap.cc:492] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.565833 12865 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.566252 12841 ts_tablet_manager.cc:1403] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.002s
I20260202 15:07:44.567525 12865 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:44.569579 12841 raft_consensus.cc:359] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.570192 12841 raft_consensus.cc:385] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.570437 12841 raft_consensus.cc:740] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.571080 12841 consensus_queue.cc:260] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.571352 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.573485 12841 ts_tablet_manager.cc:1434] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:07:44.575161 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:44.575830 12645 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:44.576195 12915 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.576586 12915 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.578040 12842 tablet_bootstrap.cc:492] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:44.578130 12915 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.578544 12842 ts_tablet_manager.cc:1403] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.077s user 0.017s sys 0.033s
I20260202 15:07:44.578810 12763 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:44.579828 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:44.580415 12794 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:44.581115 12841 tablet_bootstrap.cc:492] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.581296 12842 raft_consensus.cc:359] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.581924 12842 raft_consensus.cc:385] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.582191 12842 raft_consensus.cc:740] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.582793 12842 consensus_queue.cc:260] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.583765 13057 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:44.584110 13057 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:44.584213 12690 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:44.584498 13057 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.585659 12976 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:07:44.585954 12976 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:44.586215 12976 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:44.586886 12841 tablet_bootstrap.cc:654] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.587010 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:44.587529 12645 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:44.590428 13057 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.592413 13057 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:07:44.593711 12765 leader_election.cc:343] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.594272 12842 ts_tablet_manager.cc:1434] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.015s user 0.002s sys 0.002s
I20260202 15:07:44.594863 13057 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.595116 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:44.596516 12842 tablet_bootstrap.cc:492] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
W20260202 15:07:44.596458 12765 leader_election.cc:343] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.596907 12841 tablet_bootstrap.cc:492] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.597263 12841 ts_tablet_manager.cc:1403] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260202 15:07:44.600328 12841 raft_consensus.cc:359] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.600934 12841 raft_consensus.cc:385] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.601204 12841 raft_consensus.cc:740] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.601727 12841 consensus_queue.cc:260] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.603058 12842 tablet_bootstrap.cc:654] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.603466 12841 ts_tablet_manager.cc:1434] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260202 15:07:44.604872 12841 tablet_bootstrap.cc:492] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.607812 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:44.609872 12645 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.613027 12842 tablet_bootstrap.cc:492] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:44.610653 13059 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.613202 12841 tablet_bootstrap.cc:654] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.613413 13059 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.615753 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:44.625628 12645 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:44.626685 12763 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:44.627373 13059 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.628458 13057 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:44.628846 13057 raft_consensus.cc:697] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:44.629536 13057 consensus_queue.cc:237] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.630066 12840 tablet_bootstrap.cc:492] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:44.633067 12976 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.634917 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:44.636112 12898 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.613413 12842 ts_tablet_manager.cc:1403] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.001s
I20260202 15:07:44.630625 12840 ts_tablet_manager.cc:1403] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.007s
I20260202 15:07:44.636240 12690 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:44.638239 12976 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.636561 12898 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.641127 12898 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.641801 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.642232 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:44.642333 12722 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:44.642802 12648 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:44.643453 12615 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:44.643537 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:44.643625 12840 raft_consensus.cc:359] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.644158 12999 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:44.644486 12999 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:44.644814 12999 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:44.645282 12615 leader_election.cc:343] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.644297 12840 raft_consensus.cc:385] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.646350 12840 raft_consensus.cc:740] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.647320 12840 consensus_queue.cc:260] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.650791 13059 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:44.650360 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:44.655371 12794 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:07:44.662153 12999 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.663794 12999 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.665323 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:44.666018 12865 consensus_queue.cc:1048] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:07:44.666401 12615 leader_election.cc:343] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.667480 12842 raft_consensus.cc:359] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.668385 12842 raft_consensus.cc:385] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.668645 12842 raft_consensus.cc:740] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.669320 12842 consensus_queue.cc:260] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.680212 12840 ts_tablet_manager.cc:1434] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20260202 15:07:44.679571 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:44.681798 12724 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.681988 12842 ts_tablet_manager.cc:1434] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.045s user 0.005s sys 0.001s
I20260202 15:07:44.683106 12842 tablet_bootstrap.cc:492] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:44.685271 12841 tablet_bootstrap.cc:492] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.685742 12841 ts_tablet_manager.cc:1403] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.081s user 0.029s sys 0.029s
I20260202 15:07:44.686599 12723 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.688153 12841 raft_consensus.cc:359] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.688709 12841 raft_consensus.cc:385] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.689060 12841 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.689610 12844 consensus_queue.cc:1048] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.689640 12841 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.688750 12724 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:44.692920 12841 ts_tablet_manager.cc:1434] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260202 15:07:44.693310 12615 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:44.693737 12841 tablet_bootstrap.cc:492] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.694226 12999 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:44.696887 12999 raft_consensus.cc:697] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:44.697993 12648 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:07:44.697595 12999 consensus_queue.cc:237] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.701982 12840 tablet_bootstrap.cc:492] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:44.703361 12688 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:44.704216 12898 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.706992 12976 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:07:44.707472 12976 raft_consensus.cc:697] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:44.707296 12898 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.708369 12976 consensus_queue.cc:237] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.703835 12794 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:07:44.709503 12898 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.711010 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.715039 12546 catalog_manager.cc:5654] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
W20260202 15:07:44.718327 12615 leader_election.cc:343] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.718163 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:44.731359 12615 leader_election.cc:343] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.731700 12615 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:44.737782 12842 tablet_bootstrap.cc:654] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.740263 12898 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260202 15:07:44.749630 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 65721.2us
I20260202 15:07:44.755841 12853 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:44.757966 12840 tablet_bootstrap.cc:654] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.772589 12841 tablet_bootstrap.cc:654] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.777113 12547 catalog_manager.cc:5654] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:44.778178 12853 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.781075 12853 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.784008 12853 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.785777 12842 tablet_bootstrap.cc:492] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:44.787004 12842 ts_tablet_manager.cc:1403] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.104s user 0.013s sys 0.004s
I20260202 15:07:44.792613 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.792920 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:44.793164 12722 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:44.793491 12799 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:44.794502 12615 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:44.806351 12853 consensus_queue.cc:1048] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:44.807257 12842 raft_consensus.cc:359] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.807933 12842 raft_consensus.cc:385] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.808190 12842 raft_consensus.cc:740] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.808802 12842 consensus_queue.cc:260] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.810748 12842 ts_tablet_manager.cc:1434] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.023s user 0.004s sys 0.001s
I20260202 15:07:44.812353 12842 tablet_bootstrap.cc:492] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:44.804108 12999 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:44.815523 12544 catalog_manager.cc:5654] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 2, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:44.818897 12999 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:44.825687 12999 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.840503 12841 tablet_bootstrap.cc:492] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:44.840938 12841 ts_tablet_manager.cc:1403] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.147s user 0.007s sys 0.020s
W20260202 15:07:44.845685 12615 consensus_peers.cc:597] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Couldn't send request to peer 1eca64246cad41379cde4e2f9c3e25d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:44.849541 12842 tablet_bootstrap.cc:654] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.853662 12841 raft_consensus.cc:359] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.859201 12841 raft_consensus.cc:385] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.859400 12841 raft_consensus.cc:740] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.859737 12841 consensus_queue.cc:260] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.868191 12844 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.868623 12844 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.871668 12844 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:44.872390 12841 ts_tablet_manager.cc:1434] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.031s user 0.009s sys 0.003s
I20260202 15:07:44.870230 12898 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.873471 12898 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.874228 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.875109 12898 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:07:44.875351 12765 leader_election.cc:343] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.876461 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:44.876971 12650 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:44.879029 12841 tablet_bootstrap.cc:492] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:44.861060 12999 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.880877 12905 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.888969 12905 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.889611 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.890189 12721 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:44.890981 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:44.891587 12794 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:44.892146 12615 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:44.890664 12905 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.896281 12853 raft_consensus.cc:2804] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:44.901851 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:44.902182 12853 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:44.902511 12853 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:44.903808 12615 leader_election.cc:343] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.907855 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:44.908320 12724 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.913422 12840 tablet_bootstrap.cc:492] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:44.913803 12840 ts_tablet_manager.cc:1403] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.212s user 0.048s sys 0.027s
I20260202 15:07:44.915927 12840 raft_consensus.cc:359] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.916435 12840 raft_consensus.cc:385] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.916653 12840 raft_consensus.cc:740] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.917174 12840 consensus_queue.cc:260] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.917992 12724 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:44.903838 12763 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:44.919060 12840 ts_tablet_manager.cc:1434] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260202 15:07:44.919524 12865 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:44.919833 12865 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:44.920006 12840 tablet_bootstrap.cc:492] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:44.920143 12865 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.920406 12853 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.922031 12853 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.922935 12615 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:44.924444 12898 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:44.924465 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:44.924957 12898 raft_consensus.cc:697] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:44.925045 12723 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.904469 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:44.927906 12999 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.930841 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:44.931416 12794 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.925648 12898 consensus_queue.cc:237] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.930301 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:44.934396 12799 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.935858 12723 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:44.937829 12615 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:07:44.939440 12615 leader_election.cc:343] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:44.939744 12615 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:44.940501 12853 raft_consensus.cc:2804] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:44.942761 12853 raft_consensus.cc:697] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:44.943830 12853 consensus_queue.cc:237] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.949891 13123 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:44.958657 12842 tablet_bootstrap.cc:492] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:44.959213 12842 ts_tablet_manager.cc:1403] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.147s user 0.009s sys 0.017s
I20260202 15:07:44.959795 12840 tablet_bootstrap.cc:654] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:44.950493 13123 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.943073 12999 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:44.966624 12842 raft_consensus.cc:359] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.967211 12842 raft_consensus.cc:385] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:44.967427 12842 raft_consensus.cc:740] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:44.967952 12842 consensus_queue.cc:260] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.959141 12865 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:44.974025 12794 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:44.973949 12799 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:44.980967 12547 catalog_manager.cc:5654] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:44.988708 12842 ts_tablet_manager.cc:1434] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20260202 15:07:44.989312 12842 tablet_bootstrap.cc:492] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:44.991748 13123 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:44.994778 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:44.995239 12650 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:44.996711 12841 tablet_bootstrap.cc:654] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.008694 12865 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:45.008857 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:45.019830 13137 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.031782 13137 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.056224 12853 consensus_queue.cc:1048] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.062922 13137 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:07:45.063781 12690 leader_election.cc:343] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.065758 12547 catalog_manager.cc:5654] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:45.071591 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:45.072103 12645 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:45.045984 12999 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.073135 12999 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.074690 12999 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.076063 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:45.076746 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:45.018770 12865 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.085454 12796 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
W20260202 15:07:45.086098 12615 leader_election.cc:343] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.095325 12615 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:45.094117 12865 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.097594 12865 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:45.115499 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:45.118013 12905 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:45.118418 12905 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:45.118765 12905 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.159621 13136 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.185957 13136 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.124858 12688 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:45.188696 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:45.189188 12647 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:45.189342 13136 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.190037 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:45.193365 12615 leader_election.cc:343] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.189777 12796 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260202 15:07:45.194429 12976 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:45.196970 12976 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:45.199486 12763 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:45.201557 12844 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:45.201825 12844 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:45.202082 12844 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.198052 12976 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.203438 13059 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:07:45.207412 12615 consensus_peers.cc:597] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Couldn't send request to peer 1eca64246cad41379cde4e2f9c3e25d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:45.207299 12844 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.209045 12844 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:45.209596 12840 tablet_bootstrap.cc:492] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:45.209935 12840 ts_tablet_manager.cc:1403] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.290s user 0.004s sys 0.015s
I20260202 15:07:45.210357 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:45.213514 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:07:45.197942 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 94751.7us
I20260202 15:07:45.213892 12840 raft_consensus.cc:359] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.219853 12840 raft_consensus.cc:385] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.220117 12840 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.223009 12840 consensus_queue.cc:260] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.214968 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:45.227416 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:07:45.225003 12765 leader_election.cc:343] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:45.228615 12765 leader_election.cc:343] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.226733 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:45.229468 12723 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:45.230836 12650 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:45.230676 12976 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:45.231115 12765 leader_election.cc:343] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.231961 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:45.237569 12842 tablet_bootstrap.cc:654] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.237653 12647 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.242518 12722 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 1: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:07:45.247349 12840 ts_tablet_manager.cc:1434] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.037s user 0.001s sys 0.005s
I20260202 15:07:45.248378 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:45.242364 12976 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.252285 13123 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.252651 13123 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.253930 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:45.254277 13123 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.254443 12646 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:45.256385 12615 leader_election.cc:343] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.256206 12905 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.258742 12615 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:45.259497 12907 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:45.259786 12907 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:45.260056 12907 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.263034 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:45.263660 12649 raft_consensus.cc:2393] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 1: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:07:45.273380 12647 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:45.273636 12646 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:45.273939 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:45.274459 12690 leader_election.cc:343] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.274963 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:45.285337 12799 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.285701 12615 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:45.260079 12905 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.240751 12763 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:45.277112 12840 tablet_bootstrap.cc:492] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:45.278827 12907 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.288228 12841 tablet_bootstrap.cc:492] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:45.288735 12841 ts_tablet_manager.cc:1403] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.416s user 0.008s sys 0.006s
I20260202 15:07:45.289615 12853 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:45.291384 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:45.293980 12690 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:45.295763 13058 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:45.297001 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 57738.1us
I20260202 15:07:45.298199 12763 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:45.298902 13057 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:45.299283 13141 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:45.299338 13057 raft_consensus.cc:697] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:45.299891 13141 raft_consensus.cc:697] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:45.300002 12645 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260202 15:07:45.300284 13057 consensus_queue.cc:237] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.305933 12799 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
W20260202 15:07:45.299994 13143 log.cc:927] Time spent T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260202 15:07:45.326838 13058 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:45.328684 12840 tablet_bootstrap.cc:654] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.300695 13141 consensus_queue.cc:237] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.324158 12907 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.342903 12841 raft_consensus.cc:359] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.343657 12841 raft_consensus.cc:385] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.343935 12841 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
W20260202 15:07:45.349303 12615 leader_election.cc:343] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.346036 12841 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.353286 12844 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.353780 12615 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:45.355530 12853 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:45.356016 12853 raft_consensus.cc:697] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:45.356827 12853 consensus_queue.cc:237] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.360453 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:45.362030 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:45.362486 12724 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.352931 12688 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:45.363924 13059 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:45.364341 13059 raft_consensus.cc:697] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:45.365016 13059 consensus_queue.cc:237] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.368146 12841 ts_tablet_manager.cc:1434] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.079s user 0.006s sys 0.002s
I20260202 15:07:45.370801 13123 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:45.374949 12842 tablet_bootstrap.cc:492] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:45.375442 12842 ts_tablet_manager.cc:1403] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.386s user 0.013s sys 0.006s
I20260202 15:07:45.381867 12842 raft_consensus.cc:359] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.383627 12842 raft_consensus.cc:385] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.383940 12842 raft_consensus.cc:740] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
W20260202 15:07:45.374432 12615 leader_election.cc:343] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.382881 12844 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.384783 12842 consensus_queue.cc:260] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.386781 12841 tablet_bootstrap.cc:492] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:45.387796 12842 ts_tablet_manager.cc:1434] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20260202 15:07:45.389008 13137 consensus_queue.cc:1048] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.389472 13136 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.392403 12842 tablet_bootstrap.cc:492] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:45.410125 12842 tablet_bootstrap.cc:654] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.411896 13137 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.413934 12724 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:45.416939 12546 catalog_manager.cc:5654] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:45.424213 12842 tablet_bootstrap.cc:492] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:45.426071 12842 ts_tablet_manager.cc:1403] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.001s
I20260202 15:07:45.427702 12615 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:45.434042 12841 tablet_bootstrap.cc:654] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.435452 13137 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:45.436163 13137 raft_consensus.cc:697] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:45.437239 13137 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.439888 12842 raft_consensus.cc:359] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.440590 12842 raft_consensus.cc:385] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.440912 12842 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.441591 12842 consensus_queue.cc:260] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.446974 12842 ts_tablet_manager.cc:1434] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260202 15:07:45.447680 12842 tablet_bootstrap.cc:492] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:45.455147 12546 catalog_manager.cc:5654] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:45.456136 12842 tablet_bootstrap.cc:654] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:45.464591 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 82545.7us
I20260202 15:07:45.472097 12840 tablet_bootstrap.cc:492] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:45.472481 12840 ts_tablet_manager.cc:1403] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.196s user 0.011s sys 0.011s
I20260202 15:07:45.479054 12841 tablet_bootstrap.cc:492] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:45.479466 12841 ts_tablet_manager.cc:1403] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.093s user 0.008s sys 0.003s
I20260202 15:07:45.486402 12840 raft_consensus.cc:359] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.487223 12840 raft_consensus.cc:385] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.487502 12840 raft_consensus.cc:740] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.487586 12841 raft_consensus.cc:359] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.488197 12841 raft_consensus.cc:385] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.488431 12841 raft_consensus.cc:740] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.488925 12841 consensus_queue.cc:260] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:45.492479 12596 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:07:45.493384 12841 ts_tablet_manager.cc:1434] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260202 15:07:45.494086 12841 tablet_bootstrap.cc:492] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:45.495316 12842 tablet_bootstrap.cc:492] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:45.495683 12842 ts_tablet_manager.cc:1403] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.006s
I20260202 15:07:45.498998 12841 tablet_bootstrap.cc:654] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.498622 12842 raft_consensus.cc:359] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.499529 12842 raft_consensus.cc:385] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.499753 12842 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.500679 13059 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.501067 12853 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.501106 13059 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.502877 13059 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.506263 12546 catalog_manager.cc:5654] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:45.517999 12840 consensus_queue.cc:260] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.520062 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:45.520710 12795 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:45.521159 12840 ts_tablet_manager.cc:1434] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.048s user 0.007s sys 0.000s
I20260202 15:07:45.521960 12840 tablet_bootstrap.cc:492] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:45.522147 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:45.522794 12646 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:45.523957 12853 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.524289 12853 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.525877 12853 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.527635 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:45.527796 12690 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:45.528373 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:45.528611 13059 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:45.528991 13059 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:45.529312 13059 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.528801 12547 catalog_manager.cc:5654] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
W20260202 15:07:45.530495 12615 leader_election.cc:343] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.530303 12547 catalog_manager.cc:5654] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:45.533720 12840 tablet_bootstrap.cc:654] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.539140 13058 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.539698 13058 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.500429 12842 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.546378 13059 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.542238 13058 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.555333 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:45.557459 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:07:45.557649 12615 leader_election.cc:343] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.558005 12615 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:45.558231 12646 raft_consensus.cc:2393] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 1: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:07:45.558579 12905 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:45.559544 12905 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.562988 12688 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:45.566977 13058 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.572014 12842 ts_tablet_manager.cc:1434] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.076s user 0.005s sys 0.001s
I20260202 15:07:45.576864 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:45.577591 12647 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.584695 13141 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.585310 13141 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.585908 12647 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:45.587895 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:45.588560 12795 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.588876 12688 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:45.589821 12976 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:45.590394 12976 raft_consensus.cc:697] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:45.591395 12976 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.596100 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:45.598680 13141 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:45.599649 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:45.600983 12718 raft_consensus.cc:2393] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 1: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:07:45.602013 12765 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:45.603729 13146 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.604348 12544 catalog_manager.cc:5654] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:45.586459 13059 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.587294 12842 tablet_bootstrap.cc:492] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:45.617936 12842 tablet_bootstrap.cc:654] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.622964 12841 tablet_bootstrap.cc:492] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:45.623371 12841 ts_tablet_manager.cc:1403] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.129s user 0.035s sys 0.059s
I20260202 15:07:45.626084 12841 raft_consensus.cc:359] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.626865 12841 raft_consensus.cc:385] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.627090 12841 raft_consensus.cc:740] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.627604 12841 consensus_queue.cc:260] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.629863 12841 ts_tablet_manager.cc:1434] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:07:45.630770 12841 tablet_bootstrap.cc:492] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:45.633819 13146 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:45.638360 12841 tablet_bootstrap.cc:654] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.639828 12842 tablet_bootstrap.cc:492] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:45.640278 12842 ts_tablet_manager.cc:1403] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.006s
I20260202 15:07:45.640978 12853 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.641453 12853 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.643967 12842 raft_consensus.cc:359] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.644548 12796 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.644963 12842 raft_consensus.cc:385] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.645395 12842 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.646232 12842 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.648538 12842 ts_tablet_manager.cc:1434] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260202 15:07:45.648857 12841 tablet_bootstrap.cc:492] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:45.649227 12841 ts_tablet_manager.cc:1403] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.002s
I20260202 15:07:45.649340 12842 tablet_bootstrap.cc:492] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:45.650405 12853 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.651417 12841 raft_consensus.cc:359] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.651989 12841 raft_consensus.cc:385] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.652029 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:45.653821 12841 raft_consensus.cc:740] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.654472 12841 consensus_queue.cc:260] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.654862 12842 tablet_bootstrap.cc:654] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:45.656733 12841 ts_tablet_manager.cc:1434] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260202 15:07:45.657583 12841 tablet_bootstrap.cc:492] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
W20260202 15:07:45.659081 12615 leader_election.cc:343] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.662714 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:45.663777 12841 tablet_bootstrap.cc:654] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:45.663810 12615 leader_election.cc:343] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.664495 12615 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:45.665891 12853 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:45.674515 12795 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:45.684746 13058 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:45.696964 13146 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.697803 13146 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.701825 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:45.701992 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:45.704449 13141 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:07:45.705008 13141 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.706722 12763 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:45.707686 13155 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:45.715931 13146 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:45.720355 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:45.721697 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:45.723225 12765 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:45.724261 13141 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:45.725106 13141 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:07:45.735684 12844 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.795131 13155 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.826283 12976 consensus_queue.cc:1048] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:07:45.834177 12765 consensus_peers.cc:597] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260202 15:07:45.858152 12690 consensus_peers.cc:597] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Couldn't send request to peer 1eca64246cad41379cde4e2f9c3e25d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:45.860991 12795 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:45.889326 12905 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.889777 12905 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.892009 12905 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.907295 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:45.907968 12905 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:45.908486 12905 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:45.908718 12615 leader_election.cc:343] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:45.910643 12905 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:45.924180 12905 consensus_queue.cc:1048] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:45.939953 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:45.952692 13141 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:45.994611 12842 tablet_bootstrap.cc:492] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:45.995158 12842 ts_tablet_manager.cc:1403] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.346s user 0.106s sys 0.161s
I20260202 15:07:45.995887 12841 tablet_bootstrap.cc:492] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:45.996320 12841 ts_tablet_manager.cc:1403] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.339s user 0.095s sys 0.183s
I20260202 15:07:45.997634 12842 raft_consensus.cc:359] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.998312 12842 raft_consensus.cc:385] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.998682 12842 raft_consensus.cc:740] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.998313 12841 raft_consensus.cc:359] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:45.999442 12841 raft_consensus.cc:385] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:45.999802 12841 raft_consensus.cc:740] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:45.999432 12842 consensus_queue.cc:260] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.000361 12841 consensus_queue.cc:260] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.001829 12842 ts_tablet_manager.cc:1434] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:07:46.002378 12841 ts_tablet_manager.cc:1434] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:07:46.002717 12842 tablet_bootstrap.cc:492] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.003245 12841 tablet_bootstrap.cc:492] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:46.013736 12840 tablet_bootstrap.cc:492] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:46.014132 12840 ts_tablet_manager.cc:1403] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.492s user 0.167s sys 0.231s
I20260202 15:07:46.023726 12840 raft_consensus.cc:359] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.024324 12840 raft_consensus.cc:385] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.024554 12840 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.025058 12840 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:46.027680 12615 leader_election.cc:343] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.030678 12841 tablet_bootstrap.cc:654] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.031596 12840 ts_tablet_manager.cc:1434] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.017s user 0.002s sys 0.006s
I20260202 15:07:46.032441 12840 tablet_bootstrap.cc:492] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:46.033558 12842 tablet_bootstrap.cc:654] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.043253 12905 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:46.055590 12840 tablet_bootstrap.cc:654] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.060065 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.060384 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.077373 12976 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:46.081025 12842 tablet_bootstrap.cc:492] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:46.081530 12842 ts_tablet_manager.cc:1403] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.005s
I20260202 15:07:46.083454 12841 tablet_bootstrap.cc:492] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:46.083987 12841 ts_tablet_manager.cc:1403] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.008s
I20260202 15:07:46.083859 12842 raft_consensus.cc:359] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.084668 12842 raft_consensus.cc:385] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.084955 12842 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.085734 13059 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:46.085740 12842 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.088127 12842 ts_tablet_manager.cc:1434] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:07:46.086712 12841 raft_consensus.cc:359] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.088764 12841 raft_consensus.cc:385] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.089084 12842 tablet_bootstrap.cc:492] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.089048 12841 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.091193 12841 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.093376 12841 ts_tablet_manager.cc:1434] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260202 15:07:46.094236 12841 tablet_bootstrap.cc:492] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
W20260202 15:07:46.095316 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 53926.2us
I20260202 15:07:46.099507 12842 tablet_bootstrap.cc:654] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.113399 12841 tablet_bootstrap.cc:654] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.127337 13146 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.125922 13059 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.132261 13059 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.133893 13059 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.136370 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:46.137209 13179 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.137589 13179 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.139122 13179 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.139200 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.139717 12645 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:46.141522 12688 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:46.134676 13146 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.142346 13059 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:46.142774 13059 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:46.143146 13059 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.143280 13146 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.166776 12842 tablet_bootstrap.cc:492] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:46.167194 12842 ts_tablet_manager.cc:1403] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.019s
W20260202 15:07:46.168517 12690 leader_election.cc:343] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.171800 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.172345 12647 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:46.174317 12763 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:46.174994 13146 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:46.175346 13146 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:46.175657 13146 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.176883 13059 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.181876 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:07:46.183079 12765 leader_election.cc:343] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.168535 12840 tablet_bootstrap.cc:492] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:46.185196 12840 ts_tablet_manager.cc:1403] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.153s user 0.019s sys 0.033s
I20260202 15:07:46.191282 12842 raft_consensus.cc:359] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.193030 13146 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.193730 13141 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.193819 12842 raft_consensus.cc:385] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.194229 13141 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.195492 12842 raft_consensus.cc:740] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.199043 13141 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.199080 13146 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.196133 12842 consensus_queue.cc:260] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.201829 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.203581 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:46.203715 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:07:46.204664 12765 leader_election.cc:343] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.206317 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:46.206954 12645 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.206877 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:46.207489 12647 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:46.208297 12690 leader_election.cc:343] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.206267 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.210510 12842 ts_tablet_manager.cc:1434] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.043s user 0.005s sys 0.001s
I20260202 15:07:46.211225 12842 tablet_bootstrap.cc:492] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.196142 13059 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.219017 12763 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:46.221947 12840 raft_consensus.cc:359] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.224741 12840 raft_consensus.cc:385] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.224758 13146 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:46.225026 12840 raft_consensus.cc:740] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
W20260202 15:07:46.233258 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 95787.6us
I20260202 15:07:46.233148 12840 consensus_queue.cc:260] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.234172 12841 tablet_bootstrap.cc:492] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:46.234555 12841 ts_tablet_manager.cc:1403] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.006s
I20260202 15:07:46.236534 12853 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:46.236799 12841 raft_consensus.cc:359] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.237363 12841 raft_consensus.cc:385] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.237584 12841 raft_consensus.cc:740] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
W20260202 15:07:46.237757 12615 consensus_peers.cc:597] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:46.238090 12841 consensus_queue.cc:260] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:46.239068 12615 leader_election.cc:343] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.239398 12615 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:46.239902 12853 raft_consensus.cc:2749] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:46.240221 12841 ts_tablet_manager.cc:1434] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260202 15:07:46.240695 12615 leader_election.cc:343] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.241084 12615 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:46.241659 12853 raft_consensus.cc:2749] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:46.243466 12842 tablet_bootstrap.cc:654] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.241099 12841 tablet_bootstrap.cc:492] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:46.250415 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.251358 12841 tablet_bootstrap.cc:654] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.252663 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:46.255635 12840 ts_tablet_manager.cc:1434] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.070s user 0.005s sys 0.001s
I20260202 15:07:46.224026 12647 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:46.224112 12645 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:46.258607 13188 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.259176 12763 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:46.259057 13188 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.259773 13146 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:46.260181 13146 raft_consensus.cc:697] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:46.261103 13188 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.262127 12688 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:07:46.261183 13146 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.262009 12840 tablet_bootstrap.cc:492] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:46.265626 13059 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:46.266026 13059 raft_consensus.cc:697] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:46.266718 13059 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:46.277765 12615 leader_election.cc:343] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:46.284552 12615 leader_election.cc:343] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.284993 12615 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:46.285794 12853 raft_consensus.cc:2749] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:46.304088 12840 tablet_bootstrap.cc:654] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.291028 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:46.311760 12841 tablet_bootstrap.cc:492] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:46.312235 12841 ts_tablet_manager.cc:1403] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.013s
I20260202 15:07:46.312923 13141 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.325524 13141 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.348268 13141 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.350167 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.350754 12724 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:46.350088 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.351820 12841 raft_consensus.cc:359] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.352092 12645 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:46.352376 12841 raft_consensus.cc:385] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.352604 12841 raft_consensus.cc:740] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.353098 12841 consensus_queue.cc:260] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.351794 12765 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:46.355238 12841 ts_tablet_manager.cc:1434] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20260202 15:07:46.355091 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.356135 12841 tablet_bootstrap.cc:492] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:46.359889 12842 tablet_bootstrap.cc:492] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:46.360450 12842 ts_tablet_manager.cc:1403] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.149s user 0.012s sys 0.022s
I20260202 15:07:46.363816 13141 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:46.366061 13141 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:46.366331 13141 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.380759 12842 raft_consensus.cc:359] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.381407 12842 raft_consensus.cc:385] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.381644 12842 raft_consensus.cc:740] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.382838 12842 consensus_queue.cc:260] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:46.383210 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 96790.3us
I20260202 15:07:46.383723 12547 catalog_manager.cc:5654] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:46.385211 12840 tablet_bootstrap.cc:492] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:46.385571 12840 ts_tablet_manager.cc:1403] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.124s user 0.016s sys 0.035s
I20260202 15:07:46.384786 12842 ts_tablet_manager.cc:1434] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260202 15:07:46.386987 12842 tablet_bootstrap.cc:492] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.389721 12840 raft_consensus.cc:359] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.390303 12842 tablet_bootstrap.cc:654] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.390506 12840 raft_consensus.cc:385] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.390933 12840 raft_consensus.cc:740] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.391690 12840 consensus_queue.cc:260] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.394176 12840 ts_tablet_manager.cc:1434] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260202 15:07:46.395172 12840 tablet_bootstrap.cc:492] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:46.396477 12842 tablet_bootstrap.cc:492] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:46.396833 12842 ts_tablet_manager.cc:1403] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260202 15:07:46.399155 12842 raft_consensus.cc:359] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.399680 12842 raft_consensus.cc:385] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.400005 12842 raft_consensus.cc:740] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.399622 12546 catalog_manager.cc:5654] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:46.406574 12842 consensus_queue.cc:260] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:46.407732 12615 leader_election.cc:343] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.412707 12841 tablet_bootstrap.cc:654] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.409119 13141 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.416040 13141 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.416874 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:46.417488 12647 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:46.419291 12615 leader_election.cc:343] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.419636 12615 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:46.420106 12853 raft_consensus.cc:2749] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:46.432041 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:46.432794 12723 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.423260 12842 ts_tablet_manager.cc:1434] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260202 15:07:46.443908 12842 tablet_bootstrap.cc:492] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.456075 12723 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:46.457265 12765 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:46.458336 12842 tablet_bootstrap.cc:654] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.458801 13146 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:46.460063 13146 raft_consensus.cc:697] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:46.460939 13146 consensus_queue.cc:237] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.461688 12647 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:46.463418 13187 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.463930 13187 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.465970 13187 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.467394 12840 tablet_bootstrap.cc:654] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.472937 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.471350 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:46.478749 12615 leader_election.cc:343] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.471935 12796 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:07:46.481318 12615 leader_election.cc:343] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.481945 12615 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:46.482859 12853 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:46.493579 12841 tablet_bootstrap.cc:492] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:46.511286 12841 ts_tablet_manager.cc:1403] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.155s user 0.011s sys 0.007s
I20260202 15:07:46.513649 12841 raft_consensus.cc:359] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.514322 12841 raft_consensus.cc:385] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.514621 12841 raft_consensus.cc:740] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.515259 12841 consensus_queue.cc:260] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.516242 12842 tablet_bootstrap.cc:492] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:46.519624 12842 ts_tablet_manager.cc:1403] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.003s
I20260202 15:07:46.521682 12842 raft_consensus.cc:359] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.522099 12842 raft_consensus.cc:385] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.522305 12842 raft_consensus.cc:740] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.522838 12842 consensus_queue.cc:260] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.524343 12842 ts_tablet_manager.cc:1434] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260202 15:07:46.525003 12842 tablet_bootstrap.cc:492] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.525091 13059 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.525466 13059 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.527148 13059 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.528548 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:46.529142 12796 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:46.530150 12690 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:46.530687 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.531263 13059 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:46.531634 13059 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:46.531931 13059 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.531268 12647 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:46.534626 12840 tablet_bootstrap.cc:492] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:46.535055 12840 ts_tablet_manager.cc:1403] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.140s user 0.009s sys 0.004s
I20260202 15:07:46.536015 12546 catalog_manager.cc:5654] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:46.537396 12840 raft_consensus.cc:359] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.538051 12840 raft_consensus.cc:385] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.538290 13059 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.539702 12841 ts_tablet_manager.cc:1434] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20260202 15:07:46.540578 12841 tablet_bootstrap.cc:492] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:46.541232 13059 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.543385 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:46.543943 12647 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.545445 12841 tablet_bootstrap.cc:654] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.550539 12647 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:46.555730 12688 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:46.556509 13059 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:46.557431 13059 raft_consensus.cc:697] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:46.558210 13059 consensus_queue.cc:237] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.559849 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:46.565819 12796 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.565982 12842 tablet_bootstrap.cc:654] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.538352 12840 raft_consensus.cc:740] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.571904 12840 consensus_queue.cc:260] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.573455 12840 ts_tablet_manager.cc:1434] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.038s user 0.003s sys 0.001s
I20260202 15:07:46.574234 12840 tablet_bootstrap.cc:492] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:46.576737 12796 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:46.582160 12840 tablet_bootstrap.cc:654] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.582835 12841 tablet_bootstrap.cc:492] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:46.583200 12841 ts_tablet_manager.cc:1403] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.004s
I20260202 15:07:46.590759 12841 raft_consensus.cc:359] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.591459 12841 raft_consensus.cc:385] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.593145 12842 tablet_bootstrap.cc:492] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:46.597401 12842 ts_tablet_manager.cc:1403] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.007s
I20260202 15:07:46.599754 12842 raft_consensus.cc:359] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.600306 12842 raft_consensus.cc:385] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.600534 12842 raft_consensus.cc:740] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.601075 12842 consensus_queue.cc:260] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.595274 12841 raft_consensus.cc:740] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.602068 12841 consensus_queue.cc:260] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.603464 12841 ts_tablet_manager.cc:1434] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.020s user 0.002s sys 0.003s
I20260202 15:07:46.604944 12842 ts_tablet_manager.cc:1434] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260202 15:07:46.605659 12842 tablet_bootstrap.cc:492] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.605875 12547 catalog_manager.cc:5654] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:46.610466 12842 tablet_bootstrap.cc:654] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.611318 12841 tablet_bootstrap.cc:492] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:46.617085 12841 tablet_bootstrap.cc:654] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.630529 12841 tablet_bootstrap.cc:492] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:46.630950 12841 ts_tablet_manager.cc:1403] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.002s
I20260202 15:07:46.634670 12840 tablet_bootstrap.cc:492] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:46.635088 12840 ts_tablet_manager.cc:1403] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.061s user 0.019s sys 0.007s
I20260202 15:07:46.640869 12841 raft_consensus.cc:359] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.641532 12841 raft_consensus.cc:385] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.641755 12841 raft_consensus.cc:740] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
W20260202 15:07:46.641897 12596 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260202 15:07:46.638302 12840 raft_consensus.cc:359] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.642539 12840 raft_consensus.cc:385] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.642845 12840 raft_consensus.cc:740] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.643487 12840 consensus_queue.cc:260] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.644641 12842 tablet_bootstrap.cc:492] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:46.644995 12842 ts_tablet_manager.cc:1403] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.002s
I20260202 15:07:46.645395 12840 ts_tablet_manager.cc:1434] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260202 15:07:46.643273 12841 consensus_queue.cc:260] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.646541 12840 tablet_bootstrap.cc:492] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:46.655066 12842 raft_consensus.cc:359] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.655699 12842 raft_consensus.cc:385] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.655954 12842 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.656589 12842 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.659770 12841 ts_tablet_manager.cc:1434] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.029s user 0.004s sys 0.002s
I20260202 15:07:46.660584 12841 tablet_bootstrap.cc:492] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:46.662328 13146 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.662746 13146 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.664409 13146 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.664996 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.665396 12841 tablet_bootstrap.cc:654] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.665596 12647 raft_consensus.cc:2393] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 1: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:07:46.665671 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.666231 12842 ts_tablet_manager.cc:1434] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260202 15:07:46.666968 12842 tablet_bootstrap.cc:492] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.667030 12765 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:46.668037 13146 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.677013 12842 tablet_bootstrap.cc:654] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.679427 12840 tablet_bootstrap.cc:654] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.674558 13146 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:46.686120 12840 tablet_bootstrap.cc:492] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:46.686502 12840 ts_tablet_manager.cc:1403] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.000s
I20260202 15:07:46.688529 12840 raft_consensus.cc:359] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.689167 12840 raft_consensus.cc:385] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.689429 12840 raft_consensus.cc:740] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.690073 12840 consensus_queue.cc:260] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.693969 13123 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.694442 13123 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.708316 13123 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.719347 12840 ts_tablet_manager.cc:1434] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260202 15:07:46.720234 12840 tablet_bootstrap.cc:492] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:46.720907 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:46.720273 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.731948 12688 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:46.732281 12842 tablet_bootstrap.cc:492] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:46.732661 12842 ts_tablet_manager.cc:1403] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.066s user 0.025s sys 0.012s
I20260202 15:07:46.732807 13123 raft_consensus.cc:2749] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:46.735494 12842 raft_consensus.cc:359] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:46.735963 12765 consensus_peers.cc:597] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:46.736092 12842 raft_consensus.cc:385] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.736445 12842 raft_consensus.cc:740] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.736542 13146 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.736986 13146 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.737032 12842 consensus_queue.cc:260] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.738729 13146 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.739367 12842 ts_tablet_manager.cc:1434] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260202 15:07:46.739431 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.740310 12842 tablet_bootstrap.cc:492] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.746843 12646 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:46.749390 12763 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:46.751220 12840 tablet_bootstrap.cc:654] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.754680 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.756011 12841 tablet_bootstrap.cc:492] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:46.756397 12841 ts_tablet_manager.cc:1403] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.096s user 0.020s sys 0.028s
I20260202 15:07:46.756778 13141 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:46.757822 13141 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:46.758137 13141 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.759361 12841 raft_consensus.cc:359] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.759994 12841 raft_consensus.cc:385] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.760294 12841 raft_consensus.cc:740] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
W20260202 15:07:46.759598 12765 leader_election.cc:343] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.760917 12841 consensus_queue.cc:260] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.762962 12841 ts_tablet_manager.cc:1434] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260202 15:07:46.764312 12841 tablet_bootstrap.cc:492] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: Bootstrap starting.
I20260202 15:07:46.766287 12840 tablet_bootstrap.cc:492] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:46.766651 12840 ts_tablet_manager.cc:1403] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.000s
I20260202 15:07:46.766773 13218 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.767191 13218 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.769325 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.769852 12645 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:46.770319 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.770936 12723 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:46.770967 12763 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:46.771734 13146 raft_consensus.cc:2804] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:46.770453 13141 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.772100 13146 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:46.776919 13146 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.772136 13218 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.768762 12840 raft_consensus.cc:359] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.778285 12840 raft_consensus.cc:385] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.778518 12840 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.778939 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:46.779289 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:46.779186 12840 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:46.780524 12765 leader_election.cc:343] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.781064 13141 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.776608 12842 tablet_bootstrap.cc:654] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.779445 12645 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.781760 12840 ts_tablet_manager.cc:1434] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20260202 15:07:46.774767 12841 tablet_bootstrap.cc:654] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.788105 12645 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:46.789019 12763 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:46.789206 13146 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.790079 13141 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:46.790813 13141 raft_consensus.cc:697] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:46.791592 13141 consensus_queue.cc:237] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.794927 12840 tablet_bootstrap.cc:492] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:46.799544 13146 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.800189 12840 tablet_bootstrap.cc:654] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.800602 12842 tablet_bootstrap.cc:492] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:46.801061 12842 ts_tablet_manager.cc:1403] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.004s
I20260202 15:07:46.803422 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:46.803445 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:46.804075 12645 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.804446 12723 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.808143 13187 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.808610 13187 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.810156 13187 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.812335 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.813248 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:46.813756 12795 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
W20260202 15:07:46.815011 12615 leader_election.cc:343] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.815508 12615 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:46.815794 12841 tablet_bootstrap.cc:492] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: No bootstrap required, opened a new log
I20260202 15:07:46.816167 12841 ts_tablet_manager.cc:1403] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.006s
I20260202 15:07:46.816936 13179 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:46.817245 13179 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:46.817512 13179 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.818604 12841 raft_consensus.cc:359] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.819180 12841 raft_consensus.cc:385] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.819445 12841 raft_consensus.cc:740] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.820083 12841 consensus_queue.cc:260] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.822014 12723 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:46.823024 12765 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:46.823819 13146 raft_consensus.cc:2804] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:46.824229 13146 raft_consensus.cc:697] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:46.824990 13146 consensus_queue.cc:237] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.826262 12645 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:46.828153 12841 ts_tablet_manager.cc:1434] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260202 15:07:46.827898 13179 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.829879 13179 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.831710 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:46.832332 12795 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.832569 12840 tablet_bootstrap.cc:492] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:46.833004 12840 ts_tablet_manager.cc:1403] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.001s
I20260202 15:07:46.837415 12840 raft_consensus.cc:359] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.837970 12840 raft_consensus.cc:385] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.838208 12840 raft_consensus.cc:740] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.838806 13123 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:46.840767 12795 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:46.841763 12615 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:46.842410 13187 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:46.842844 13187 raft_consensus.cc:697] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:46.838723 12840 consensus_queue.cc:260] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.889006 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:46.900655 13218 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.904525 13188 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.928678 13218 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.925194 12842 raft_consensus.cc:359] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.929490 12842 raft_consensus.cc:385] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.929730 13190 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:46.929819 12842 raft_consensus.cc:740] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.843534 13187 consensus_queue.cc:237] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.928709 13188 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.936085 12842 consensus_queue.cc:260] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.936986 13188 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.909333 12547 catalog_manager.cc:5654] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:46.930142 13190 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.944268 13190 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:46.930480 12840 ts_tablet_manager.cc:1434] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.097s user 0.012s sys 0.009s
I20260202 15:07:46.930392 13218 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:46.927376 13141 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:46.947579 12840 tablet_bootstrap.cc:492] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:46.950259 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.954393 12840 tablet_bootstrap.cc:654] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:46.955905 12615 leader_election.cc:343] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.958528 12842 ts_tablet_manager.cc:1434] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.157s user 0.003s sys 0.007s
I20260202 15:07:46.959961 12842 tablet_bootstrap.cc:492] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:46.961359 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.966200 12840 tablet_bootstrap.cc:492] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:46.965860 12547 catalog_manager.cc:5654] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:46.966735 12840 ts_tablet_manager.cc:1403] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.004s
W20260202 15:07:46.970342 12615 leader_election.cc:343] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:46.975543 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:46.934062 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:46.978957 12647 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:46.977543 12840 raft_consensus.cc:359] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.979393 12840 raft_consensus.cc:385] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:46.979622 12840 raft_consensus.cc:740] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:46.979748 12763 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:46.980140 12840 consensus_queue.cc:260] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:46.980625 13057 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:46.981017 13057 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:46.981345 13057 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:46.985316 12840 ts_tablet_manager.cc:1434] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20260202 15:07:46.986060 12840 tablet_bootstrap.cc:492] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:46.990901 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:46.996443 12840 tablet_bootstrap.cc:654] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:46.996665 12842 tablet_bootstrap.cc:654] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:47.007855 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:47.008378 12795 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 0.
I20260202 15:07:47.009126 13146 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:07:47.025636 12765 leader_election.cc:343] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:47.049118 12546 catalog_manager.cc:5654] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:47.053146 13057 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.054843 13057 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:47.055330 12723 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.055395 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:47.055948 12646 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.061542 12615 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:47.066823 13190 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:47.069859 13190 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.065667 13179 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:47.071585 13179 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:47.071868 13179 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.074252 13218 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:47.077736 13179 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.079955 13179 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:47.080941 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:47.081430 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:07:47.081700 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:47.082873 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:07:47.084942 12615 leader_election.cc:343] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:47.090546 12795 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.079913 13190 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:47.048383 12690 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:47.111145 12840 tablet_bootstrap.cc:492] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:47.092402 12648 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:47.111743 12840 ts_tablet_manager.cc:1403] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.126s user 0.013s sys 0.020s
W20260202 15:07:47.100715 12765 leader_election.cc:343] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:47.119277 13229 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:47.120810 12688 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:47.121821 13229 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:47.122120 13229 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:47.122399 13229 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.126118 12795 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:07:47.122537 12840 raft_consensus.cc:359] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.128885 12840 raft_consensus.cc:385] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:47.129093 12840 raft_consensus.cc:740] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:47.129585 12840 consensus_queue.cc:260] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.131549 12615 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:47.109102 13190 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:47.111737 12646 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:07:47.135807 13188 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:47.139807 13188 raft_consensus.cc:697] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:07:47.140197 12763 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:47.140503 12840 ts_tablet_manager.cc:1434] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
W20260202 15:07:47.114382 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 70265us
I20260202 15:07:47.142150 12840 tablet_bootstrap.cc:492] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:47.142170 13146 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:47.137651 12722 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.140518 13188 consensus_queue.cc:237] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.150305 12842 tablet_bootstrap.cc:492] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:47.148622 13146 raft_consensus.cc:697] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:07:47.151180 12842 ts_tablet_manager.cc:1403] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.191s user 0.009s sys 0.036s
I20260202 15:07:47.147384 13229 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.160712 13123 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:47.152388 13146 consensus_queue.cc:237] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.165328 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:47.165896 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:47.166393 12647 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.169430 12842 raft_consensus.cc:359] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.170320 12842 raft_consensus.cc:385] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:47.170572 12842 raft_consensus.cc:740] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:47.172756 13229 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:47.173944 12647 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:47.174185 12842 consensus_queue.cc:260] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.175096 12688 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:47.175922 13059 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:47.176303 13059 raft_consensus.cc:697] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:47.176780 12842 ts_tablet_manager.cc:1434] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.025s user 0.000s sys 0.004s
I20260202 15:07:47.177521 12842 tablet_bootstrap.cc:492] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:47.177600 13197 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:47.178014 13197 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.180030 13197 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:47.180583 13059 consensus_queue.cc:237] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.185132 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:47.186265 12615 leader_election.cc:343] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:47.182524 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:47.258126 12842 tablet_bootstrap.cc:654] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:47.260128 13123 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:47.227627 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:47.277385 13123 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:47.195521 12690 leader_election.cc:343] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:47.278537 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 83932.4us
I20260202 15:07:47.279007 13123 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:47.279446 13123 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:07:47.279803 12690 leader_election.cc:343] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
W20260202 15:07:47.282480 12615 leader_election.cc:343] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:47.230258 13231 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:47.282933 12615 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:47.283409 13231 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.289167 12547 catalog_manager.cc:5654] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:07:47.308682 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:47.286834 13197 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:47.309235 12647 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:47.313179 13231 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:47.313560 12840 tablet_bootstrap.cc:654] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:47.319517 12688 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:47.321928 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:47.326392 13123 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:47.344000 12546 catalog_manager.cc:5654] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:47.340016 13123 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:47.330513 13057 consensus_queue.cc:1048] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
W20260202 15:07:47.424530 13244 log.cc:927] Time spent T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20260202 15:07:47.424226 13123 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.423913 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:47.446285 12800 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
W20260202 15:07:47.449811 13246 log.cc:927] Time spent T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
W20260202 15:07:47.460769 13247 log.cc:927] Time spent T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
W20260202 15:07:47.494346 12615 consensus_peers.cc:597] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:47.461256 13229 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:07:47.479225 13245 log.cc:927] Time spent T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.132s user 0.000s sys 0.003s
I20260202 15:07:47.533613 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:07:47.534229 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 57632.8us
I20260202 15:07:47.538182 12840 tablet_bootstrap.cc:492] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:47.538738 12840 ts_tablet_manager.cc:1403] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.397s user 0.022s sys 0.007s
W20260202 15:07:47.542718 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 43928.5us
I20260202 15:07:47.515309 13229 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:47.566144 13250 log.cc:927] Time spent T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.221s user 0.001s sys 0.000s
I20260202 15:07:47.584290 12842 tablet_bootstrap.cc:492] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:47.584663 12842 ts_tablet_manager.cc:1403] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.407s user 0.024s sys 0.028s
I20260202 15:07:47.591675 13179 consensus_queue.cc:1048] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:47.592814 12840 raft_consensus.cc:359] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.593436 12840 raft_consensus.cc:385] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:47.593681 12840 raft_consensus.cc:740] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:47.593559 12547 catalog_manager.cc:5654] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:47.594213 12840 consensus_queue.cc:260] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.613765 12840 ts_tablet_manager.cc:1434] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.075s user 0.003s sys 0.002s
I20260202 15:07:47.616664 13229 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:47.596589 13123 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.622133 12840 tablet_bootstrap.cc:492] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:47.585676 13218 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
W20260202 15:07:47.629021 12615 consensus_peers.cc:597] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:47.635036 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:47.637043 12650 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:47.638715 12688 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:47.663292 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:47.656680 13123 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:47.656741 13231 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:47.669517 12842 raft_consensus.cc:359] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.671393 12842 raft_consensus.cc:385] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:47.671691 12842 raft_consensus.cc:740] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:47.672230 12842 consensus_queue.cc:260] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.672962 13231 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:47.673765 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:47.674150 13231 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.674319 12795 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.686002 12842 ts_tablet_manager.cc:1434] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.101s user 0.003s sys 0.003s
I20260202 15:07:47.686668 12842 tablet_bootstrap.cc:492] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:47.689759 12690 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:47.697080 13123 raft_consensus.cc:2749] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:47.664255 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:47.697860 12641 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.753194 12842 tablet_bootstrap.cc:654] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:47.768760 12840 tablet_bootstrap.cc:654] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
W20260202 15:07:47.775856 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 155901us
W20260202 15:07:47.776532 12765 consensus_peers.cc:597] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260202 15:07:47.781679 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 91997.2us
I20260202 15:07:47.784687 13231 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.787534 13231 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:07:47.793838 12690 leader_election.cc:343] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Illegal state: must be running to vote when last-logged opid is not known
I20260202 15:07:47.795902 12842 tablet_bootstrap.cc:492] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:47.796366 12842 ts_tablet_manager.cc:1403] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.110s user 0.008s sys 0.002s
I20260202 15:07:47.764516 12641 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:47.800547 12842 raft_consensus.cc:359] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.801318 12842 raft_consensus.cc:385] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:47.801657 12842 raft_consensus.cc:740] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:47.814810 12842 consensus_queue.cc:260] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.803493 12795 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:47.818585 12840 tablet_bootstrap.cc:492] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:47.818640 12842 ts_tablet_manager.cc:1434] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.022s user 0.003s sys 0.004s
I20260202 15:07:47.819023 12840 ts_tablet_manager.cc:1403] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.197s user 0.008s sys 0.005s
I20260202 15:07:47.819619 12842 tablet_bootstrap.cc:492] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2: Bootstrap starting.
I20260202 15:07:47.821409 12840 raft_consensus.cc:359] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.821964 12840 raft_consensus.cc:385] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:47.822209 12840 raft_consensus.cc:740] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:47.822753 12840 consensus_queue.cc:260] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.823803 12795 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.824186 12840 ts_tablet_manager.cc:1434] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:07:47.824750 12842 tablet_bootstrap.cc:654] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:47.824944 12840 tablet_bootstrap.cc:492] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:47.832775 12688 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:07:47.833532 13257 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:47.834543 13257 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:47.834928 13257 raft_consensus.cc:697] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:47.834883 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:47.835477 12791 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.835850 13257 consensus_queue.cc:237] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.847208 12791 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:47.853005 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:47.853504 12641 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.853673 12690 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:47.855141 13258 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:47.855528 13258 raft_consensus.cc:697] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:47.856202 13258 consensus_queue.cc:237] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:47.867301 13256 consensus_queue.cc:1048] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:47.874243 12722 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:47.899585 12842 tablet_bootstrap.cc:492] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2: No bootstrap required, opened a new log
I20260202 15:07:47.900017 12842 ts_tablet_manager.cc:1403] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2: Time spent bootstrapping tablet: real 0.081s user 0.019s sys 0.025s
I20260202 15:07:47.862484 12641 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:47.945827 13261 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
W20260202 15:07:47.950466 12765 consensus_peers.cc:597] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Couldn't send request to peer 6921687ed3814cccb2bd9c606875b9f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:47.957373 12547 catalog_manager.cc:5654] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:47.960506 12840 tablet_bootstrap.cc:654] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:47.963142 12547 catalog_manager.cc:5654] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:47.970168 13252 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260202 15:07:47.972146 13146 consensus_queue.cc:1048] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260202 15:07:48.070541 12842 raft_consensus.cc:359] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.072913 12842 raft_consensus.cc:385] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:48.073122 12842 raft_consensus.cc:740] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Initialized, Role: FOLLOWER
I20260202 15:07:48.073536 12842 consensus_queue.cc:260] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:48.071095 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 58913us
W20260202 15:07:48.079386 12596 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260202 15:07:48.084932 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 103792us
I20260202 15:07:48.093470 13190 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:48.099705 13190 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.102092 13190 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:48.113725 12842 ts_tablet_manager.cc:1434] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2: Time spent starting tablet: real 0.213s user 0.007s sys 0.000s
W20260202 15:07:48.164902 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 55139.9us
I20260202 15:07:48.170576 12717 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:48.171780 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:48.206323 13232 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:48.207260 13232 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.211625 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:48.246542 12690 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:48.250193 13146 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:07:48.251058 13146 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.257131 13232 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:48.265105 13256 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:48.266023 13256 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.281615 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:48.282333 12650 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:48.289212 13258 consensus_queue.cc:1048] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260202 15:07:48.270452 13256 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:48.345196 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:48.345818 12791 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 0.
I20260202 15:07:48.350373 12840 tablet_bootstrap.cc:492] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:48.352376 13283 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260202 15:07:48.356308 12840 ts_tablet_manager.cc:1403] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.532s user 0.020s sys 0.011s
I20260202 15:07:48.352840 13232 consensus_queue.cc:1048] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260202 15:07:48.374302 12690 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:07:48.375921 13251 raft_consensus.cc:2804] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:07:48.376405 13251 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:07:48.376814 13251 raft_consensus.cc:3060] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:48.439904 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:48.442279 12840 raft_consensus.cc:359] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.457602 13190 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:48.458314 12840 raft_consensus.cc:385] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:48.447085 12644 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 0.
I20260202 15:07:48.460266 12840 raft_consensus.cc:740] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:48.464718 12840 consensus_queue.cc:260] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:07:48.465587 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 163601us
I20260202 15:07:48.473774 13251 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.481696 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:48.482359 12724 raft_consensus.cc:2393] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 1: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:07:48.482961 13251 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:48.484193 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:07:48.484778 12800 raft_consensus.cc:3060] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:48.503194 12840 ts_tablet_manager.cc:1434] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.146s user 0.004s sys 0.004s
I20260202 15:07:48.503937 12800 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:48.506443 12690 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:07:48.490350 13296 log.cc:927] Time spent T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.143s user 0.004s sys 0.010s
I20260202 15:07:48.533360 12840 tablet_bootstrap.cc:492] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:48.572295 13264 raft_consensus.cc:2804] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:07:48.572628 13146 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:48.572889 13264 raft_consensus.cc:697] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:07:48.573894 13264 consensus_queue.cc:237] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.599009 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:07:48.599464 12650 raft_consensus.cc:3060] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:48.611830 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:07:48.613271 12763 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:07:48.614488 13146 raft_consensus.cc:2764] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260202 15:07:48.625799 13123 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:48.633314 12650 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:07:48.634279 12840 tablet_bootstrap.cc:654] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:48.640275 12840 tablet_bootstrap.cc:492] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:48.640690 12840 ts_tablet_manager.cc:1403] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.108s user 0.010s sys 0.000s
I20260202 15:07:48.642194 13232 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:07:48.642601 13232 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.643117 12840 raft_consensus.cc:359] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.643676 12840 raft_consensus.cc:385] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:48.644731 12840 raft_consensus.cc:740] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:48.645249 12840 consensus_queue.cc:260] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.646906 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:48.648766 13232 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:48.649046 12840 ts_tablet_manager.cc:1434] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260202 15:07:48.649271 12763 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:48.649947 12840 tablet_bootstrap.cc:492] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0: Bootstrap starting.
I20260202 15:07:48.651021 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:07:48.652878 13248 raft_consensus.cc:2749] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:07:48.668027 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:07:48.671900 12840 tablet_bootstrap.cc:654] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0: Neither blocks nor log segments found. Creating new log.
I20260202 15:07:48.698895 12547 catalog_manager.cc:5654] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:48.705971 12690 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:07:48.711401 13232 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:07:48.734052 12725 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:48.794009 12722 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:48.817463 12840 tablet_bootstrap.cc:492] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0: No bootstrap required, opened a new log
I20260202 15:07:48.817873 12840 ts_tablet_manager.cc:1403] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0: Time spent bootstrapping tablet: real 0.168s user 0.049s sys 0.083s
I20260202 15:07:48.820309 12840 raft_consensus.cc:359] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.821111 12840 raft_consensus.cc:385] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:07:48.821295 12840 raft_consensus.cc:740] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Initialized, Role: FOLLOWER
I20260202 15:07:48.821766 12840 consensus_queue.cc:260] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:07:48.824417 12840 ts_tablet_manager.cc:1434] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260202 15:07:48.844745 12722 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:07:49.048190 13232 consensus_queue.cc:1048] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:49.057704 13251 consensus_queue.cc:1048] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:07:49.834110 11451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:07:49.852854 13317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:49.853070 13318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:07:49.854849 13320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:07:49.855964 11451 server_base.cc:1060] running on GCE node
I20260202 15:07:49.857151 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260202 15:07:49.857398 11451 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260202 15:07:49.857563 11451 hybrid_clock.cc:648] HybridClock initialized: now 1770044869857548 us; error 0 us; skew 500 ppm
I20260202 15:07:49.858269 11451 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260202 15:07:49.861645 11451 webserver.cc:492] Webserver started at http://127.11.46.196:34445/ using document root <none> and password file <none>
I20260202 15:07:49.862133 11451 fs_manager.cc:362] Metadata directory not provided
I20260202 15:07:49.862394 11451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:07:49.862687 11451 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:07:49.864001 11451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-3-root/instance:
uuid: "66b8c715b9e6479e9218b08bd3156380"
format_stamp: "Formatted at 2026-02-02 15:07:49 on dist-test-slave-rflp"
I20260202 15:07:49.868957 11451 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260202 15:07:49.872607 13325 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:49.873417 11451 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260202 15:07:49.873705 11451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-3-root
uuid: "66b8c715b9e6479e9218b08bd3156380"
format_stamp: "Formatted at 2026-02-02 15:07:49 on dist-test-slave-rflp"
I20260202 15:07:49.874004 11451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:07:49.889709 11451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:07:49.891376 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:07:49.893100 11451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:07:49.895201 11451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:07:49.895360 11451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:49.895558 11451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:07:49.895732 11451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:07:49.950140 11451 rpc_server.cc:307] RPC server started. Bound to: 127.11.46.196:36191
I20260202 15:07:49.950270 13387 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.46.196:36191 every 8 connection(s)
I20260202 15:07:49.955027 11451 auto_leader_rebalancer-test.cc:417] add a tserver: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:49.966770 13388 heartbeater.cc:344] Connected to a master server at 127.11.46.254:37385
I20260202 15:07:49.967182 13388 heartbeater.cc:461] Registering TS with master...
I20260202 15:07:49.968012 13388 heartbeater.cc:507] Master 127.11.46.254:37385 requested a full tablet report, sending...
I20260202 15:07:49.969673 12546 ts_manager.cc:194] Registered new tserver with Master: 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:07:49.971016 12546 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39704
I20260202 15:07:50.196374 12795 consensus_queue.cc:237] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.202028 12641 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:50.203135 12722 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:50.204418 13395 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:50.205394 13298 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:50.213595 13395 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.215649 12725 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.232586 12546 catalog_manager.cc:5654] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260202 15:07:50.247416 12764 consensus_peers.cc:597] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb4cc1899a25452c9c57398fe4022645. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:50.252959 12645 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.289775 12722 consensus_queue.cc:237] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.299602 12795 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:50.301826 12641 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:50.303453 13258 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:50.304414 13059 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:07:50.314952 12689 consensus_peers.cc:597] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ca9793f4e4e46418e44fbdc3d3a4794. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:50.315100 13059 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.316686 12795 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.319726 12641 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.334651 12544 catalog_manager.cc:5654] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.338574 12722 consensus_queue.cc:237] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.345060 12795 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:07:50.346331 12689 consensus_peers.cc:597] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e78bd8179024ae0aca19878b91342a9. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:50.346946 13258 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:50.349773 12641 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:50.350764 13258 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:50.361609 13258 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.363070 12795 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.364594 12641 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.380271 12544 catalog_manager.cc:5654] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:50.389300 12645 consensus_queue.cc:237] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.402056 12795 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:50.404742 12722 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:50.406636 13261 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:07:50.408308 13261 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:07:50.409179 12614 consensus_peers.cc:597] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a776a09df954aa6ac5c756e8da6c56a. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:50.430095 13417 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.431906 12722 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.443321 12798 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:50.458551 12546 catalog_manager.cc:5654] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:50.754000 13421 ts_tablet_manager.cc:933] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:50.755882 13421 tablet_copy_client.cc:323] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:50.765479 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 9e78bd8179024ae0aca19878b91342a9 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:50.765954 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 9e78bd8179024ae0aca19878b91342a9 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-9e78bd8179024ae0aca19878b91342a9
I20260202 15:07:50.771997 12736 tablet_copy_source_session.cc:215] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:50.775313 13421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e78bd8179024ae0aca19878b91342a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:50.788791 13421 tablet_copy_client.cc:806] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:50.789273 13421 tablet_copy_client.cc:670] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:50.791356 13424 ts_tablet_manager.cc:933] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:07:50.792910 13421 tablet_copy_client.cc:538] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:50.793519 13424 tablet_copy_client.cc:323] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:07:50.800210 13421 tablet_bootstrap.cc:492] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:50.805053 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 5a776a09df954aa6ac5c756e8da6c56a from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:07:50.805536 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 5a776a09df954aa6ac5c756e8da6c56a from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-5a776a09df954aa6ac5c756e8da6c56a
I20260202 15:07:50.809839 12660 tablet_copy_source_session.cc:215] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:50.815378 13424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a776a09df954aa6ac5c756e8da6c56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:50.817555 13427 ts_tablet_manager.cc:933] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:50.819979 13421 tablet_bootstrap.cc:492] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:50.820950 13421 tablet_bootstrap.cc:492] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:50.822031 13421 ts_tablet_manager.cc:1403] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.012s
I20260202 15:07:50.825305 13421 raft_consensus.cc:359] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.826141 13421 raft_consensus.cc:740] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:50.827366 13421 consensus_queue.cc:260] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.836163 13424 tablet_copy_client.cc:806] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:50.836651 13424 tablet_copy_client.cc:670] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:50.840554 13388 heartbeater.cc:499] Master 127.11.46.254:37385 was elected leader, sending a full tablet report...
I20260202 15:07:50.841374 13424 tablet_copy_client.cc:538] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:50.841702 13421 ts_tablet_manager.cc:1434] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.019s user 0.003s sys 0.008s
I20260202 15:07:50.841711 13427 tablet_copy_client.cc:323] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:07:50.843081 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-9e78bd8179024ae0aca19878b91342a9 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:50.843525 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-9e78bd8179024ae0aca19878b91342a9 on tablet 9e78bd8179024ae0aca19878b91342a9 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:50.854737 13424 tablet_bootstrap.cc:492] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:50.857267 13421 ts_tablet_manager.cc:933] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:50.869059 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet cb4cc1899a25452c9c57398fe4022645 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:07:50.869525 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet cb4cc1899a25452c9c57398fe4022645 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-cb4cc1899a25452c9c57398fe4022645
I20260202 15:07:50.870172 13421 tablet_copy_client.cc:323] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:50.871917 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 6ca9793f4e4e46418e44fbdc3d3a4794 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:50.872359 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 6ca9793f4e4e46418e44fbdc3d3a4794 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-6ca9793f4e4e46418e44fbdc3d3a4794
I20260202 15:07:50.878079 12736 tablet_copy_source_session.cc:215] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:50.878646 12810 tablet_copy_source_session.cc:215] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:50.879680 13424 tablet_bootstrap.cc:492] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:50.880493 13424 tablet_bootstrap.cc:492] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:50.880636 13421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca9793f4e4e46418e44fbdc3d3a4794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:50.881170 13424 ts_tablet_manager.cc:1403] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.009s
I20260202 15:07:50.884696 13424 raft_consensus.cc:359] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.885411 13424 raft_consensus.cc:740] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:50.885926 13424 consensus_queue.cc:260] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.887027 13427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4cc1899a25452c9c57398fe4022645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:50.889943 13424 ts_tablet_manager.cc:1434] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20260202 15:07:50.893914 13421 tablet_copy_client.cc:806] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:50.894354 13421 tablet_copy_client.cc:670] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:50.899017 13421 tablet_copy_client.cc:538] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:50.901180 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-5a776a09df954aa6ac5c756e8da6c56a received from {username='slave'} at 127.0.0.1:42782
I20260202 15:07:50.901608 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-5a776a09df954aa6ac5c756e8da6c56a on tablet 5a776a09df954aa6ac5c756e8da6c56a with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:50.910320 13427 tablet_copy_client.cc:806] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:50.910943 13427 tablet_copy_client.cc:670] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:50.915485 13421 tablet_bootstrap.cc:492] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:50.916690 13427 tablet_copy_client.cc:538] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:50.925642 13427 tablet_bootstrap.cc:492] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:50.934958 13421 tablet_bootstrap.cc:492] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:50.935709 13421 tablet_bootstrap.cc:492] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:50.936250 13421 ts_tablet_manager.cc:1403] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.000s
I20260202 15:07:50.938447 13421 raft_consensus.cc:359] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.939157 13421 raft_consensus.cc:740] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:50.939694 13421 consensus_queue.cc:260] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.941967 13421 ts_tablet_manager.cc:1434] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:07:50.948689 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-6ca9793f4e4e46418e44fbdc3d3a4794 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:50.949054 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-6ca9793f4e4e46418e44fbdc3d3a4794 on tablet 6ca9793f4e4e46418e44fbdc3d3a4794 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:50.950053 13427 tablet_bootstrap.cc:492] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:50.950925 13427 tablet_bootstrap.cc:492] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:50.951548 13427 ts_tablet_manager.cc:1403] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.005s
I20260202 15:07:50.954144 13427 raft_consensus.cc:359] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.954895 13427 raft_consensus.cc:740] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:50.955430 13427 consensus_queue.cc:260] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:50.960987 13427 ts_tablet_manager.cc:1434] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20260202 15:07:50.963371 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-cb4cc1899a25452c9c57398fe4022645 received from {username='slave'} at 127.0.0.1:32824
I20260202 15:07:50.963765 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-cb4cc1899a25452c9c57398fe4022645 on tablet cb4cc1899a25452c9c57398fe4022645 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:51.197306 13363 raft_consensus.cc:1217] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:51.201450 13363 raft_consensus.cc:1217] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:51.228127 13363 raft_consensus.cc:1217] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:51.429644 13363 raft_consensus.cc:1217] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:51.627269 13410 raft_consensus.cc:1064] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:51.629400 13410 consensus_queue.cc:237] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:51.634852 12795 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.635483 13363 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.636099 12645 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.636083 13258 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.637063 13410 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.638077 13059 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.661028 13410 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.662433 12645 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.675194 13363 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.679005 12799 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.685791 12546 catalog_manager.cc:5654] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.716820 12726 consensus_queue.cc:237] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:51.723188 12641 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:51.723788 13197 raft_consensus.cc:1064] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:51.724295 13059 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:51.725768 13197 consensus_queue.cc:237] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:51.724004 13362 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:51.727561 13441 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:51.730427 13402 raft_consensus.cc:1064] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:51.733827 13402 consensus_queue.cc:237] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:51.733955 13441 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.736931 12795 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.737074 13363 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.736856 13362 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.742800 13261 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.737656 12645 raft_consensus.cc:2955] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.748801 13448 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.772852 13361 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.773626 12726 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.774310 13402 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.775704 13141 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.779453 12641 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.781629 13436 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.789966 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 9e78bd8179024ae0aca19878b91342a9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:51.781225 12547 catalog_manager.cc:5654] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.807034 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e78bd8179024ae0aca19878b91342a9 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:07:51.810349 12722 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.810005 13455 tablet_replica.cc:333] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:07:51.811579 13197 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:07:51.815943 13455 raft_consensus.cc:2243] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:51.816713 13455 raft_consensus.cc:2272] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:51.815609 13363 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.820824 13455 ts_tablet_manager.cc:1916] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:51.813913 13448 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.835006 12722 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.845221 13455 ts_tablet_manager.cc:1929] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:51.845607 13455 log.cc:1199] T 9e78bd8179024ae0aca19878b91342a9 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/9e78bd8179024ae0aca19878b91342a9
I20260202 15:07:51.852411 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 9e78bd8179024ae0aca19878b91342a9 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:51.857533 12547 catalog_manager.cc:5654] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.859805 13436 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.861426 12718 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.867818 12795 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.879886 13363 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.887012 12641 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.889043 12546 catalog_manager.cc:5654] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:07:51.894584 12650 consensus_queue.cc:237] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:51.901522 12796 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:51.902683 13261 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:51.906039 12796 consensus_queue.cc:237] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:51.906499 13363 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:51.911317 13261 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:51.915832 13197 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.920958 13362 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:51.920360 13363 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.923286 12641 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:51.922171 12796 raft_consensus.cc:2955] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.924400 13402 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:51.925088 13436 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:51.927467 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 5a776a09df954aa6ac5c756e8da6c56a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:51.932550 13141 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.945134 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet cb4cc1899a25452c9c57398fe4022645 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:51.945528 12547 catalog_manager.cc:5654] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.947858 13363 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.955794 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a776a09df954aa6ac5c756e8da6c56a with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:07:51.961483 13258 raft_consensus.cc:1064] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:51.964488 13458 tablet_replica.cc:333] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:07:51.963780 13258 consensus_queue.cc:237] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:51.965680 13458 raft_consensus.cc:2243] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:51.967358 13458 raft_consensus.cc:2272] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:51.969468 12546 catalog_manager.cc:5654] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.974355 13458 ts_tablet_manager.cc:1916] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:51.975432 12641 raft_consensus.cc:2955] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:51.978283 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet cb4cc1899a25452c9c57398fe4022645 with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:07:51.990360 12795 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.991700 13410 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.992765 12641 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.993920 13410 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:51.995227 13363 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:51.996315 13410 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:52.003449 13059 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:52.007195 12650 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:52.008812 13458 ts_tablet_manager.cc:1929] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:52.009310 13458 log.cc:1199] T 5a776a09df954aa6ac5c756e8da6c56a P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/5a776a09df954aa6ac5c756e8da6c56a
I20260202 15:07:52.010946 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet 5a776a09df954aa6ac5c756e8da6c56a (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:52.011054 13458 tablet_replica.cc:333] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:07:52.012737 13458 raft_consensus.cc:2243] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:52.013537 13458 raft_consensus.cc:2272] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:52.016291 13363 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:52.027678 12546 catalog_manager.cc:5654] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:52.028645 13458 ts_tablet_manager.cc:1916] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:52.032269 12799 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:52.044236 12726 consensus_queue.cc:237] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:52.048102 13458 ts_tablet_manager.cc:1929] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:52.048506 13458 log.cc:1199] T cb4cc1899a25452c9c57398fe4022645 P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/cb4cc1899a25452c9c57398fe4022645
I20260202 15:07:52.049731 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet cb4cc1899a25452c9c57398fe4022645 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:52.051458 13363 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:52.070310 13258 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:52.072443 12799 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:52.076771 13059 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:52.082221 13258 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:52.087319 13363 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:52.089958 12799 raft_consensus.cc:2955] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:52.095535 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 6ca9793f4e4e46418e44fbdc3d3a4794 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:52.102288 12547 catalog_manager.cc:5654] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:52.109977 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ca9793f4e4e46418e44fbdc3d3a4794 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:07:52.111590 13463 tablet_replica.cc:333] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:07:52.112403 13463 raft_consensus.cc:2243] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:52.112931 13463 raft_consensus.cc:2272] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:52.115312 13463 ts_tablet_manager.cc:1916] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:52.124111 13463 ts_tablet_manager.cc:1929] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:52.124411 13463 log.cc:1199] T 6ca9793f4e4e46418e44fbdc3d3a4794 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/6ca9793f4e4e46418e44fbdc3d3a4794
I20260202 15:07:52.125613 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet 6ca9793f4e4e46418e44fbdc3d3a4794 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:53.165269 12795 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.170001 12718 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:07:53.170082 12764 consensus_peers.cc:597] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a50a506e17f747e0a940bc3c982336c6. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:53.170341 12650 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:53.171259 13466 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:07:53.172309 13457 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:53.181473 13457 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.184227 12718 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.186086 12650 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.194344 12718 consensus_queue.cc:237] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.196857 12546 catalog_manager.cc:5654] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260202 15:07:53.203783 12689 consensus_peers.cc:597] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00b1a15d2d1246e1abe52586ddf512ef. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:53.204732 12795 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:53.206548 12645 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:53.207585 13470 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:53.209247 13258 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:53.216038 13258 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.217748 12795 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.217674 12641 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.228797 12645 consensus_queue.cc:237] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:07:53.246850 12614 consensus_peers.cc:597] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f70fb0b2ffc4a51b06fe26a22355100. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:53.248306 12795 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:53.250088 13464 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:53.246502 12544 catalog_manager.cc:5654] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.252647 12718 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:53.253710 13464 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:53.262423 12795 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.260159 13464 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.270641 12718 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.277987 12726 consensus_queue.cc:237] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.279572 12544 catalog_manager.cc:5654] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.285583 12799 raft_consensus.cc:1275] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:53.287649 13258 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:07:53.288341 12689 consensus_peers.cc:597] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65f098e59c0240a3a65cbd15dc358b64. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:53.290630 12645 raft_consensus.cc:1275] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:53.292604 13258 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:53.295794 13470 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.304979 12795 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.316633 12546 catalog_manager.cc:5654] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:53.324839 12645 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:53.668078 13483 ts_tablet_manager.cc:933] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:53.670989 13483 tablet_copy_client.cc:323] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:07:53.672573 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet a50a506e17f747e0a940bc3c982336c6 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:07:53.673010 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet a50a506e17f747e0a940bc3c982336c6 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-a50a506e17f747e0a940bc3c982336c6
I20260202 15:07:53.680141 12810 tablet_copy_source_session.cc:215] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:53.684679 13483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50a506e17f747e0a940bc3c982336c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:53.697088 13483 tablet_copy_client.cc:806] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:53.697494 13483 tablet_copy_client.cc:670] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:53.700825 13483 tablet_copy_client.cc:538] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:53.707690 13483 tablet_bootstrap.cc:492] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:53.722949 13483 tablet_bootstrap.cc:492] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:53.723646 13483 tablet_bootstrap.cc:492] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:53.724323 13483 ts_tablet_manager.cc:1403] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20260202 15:07:53.726934 13483 raft_consensus.cc:359] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.727722 13483 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:53.728282 13483 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.731849 13487 ts_tablet_manager.cc:933] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:53.732555 13483 ts_tablet_manager.cc:1434] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.008s user 0.001s sys 0.004s
I20260202 15:07:53.733922 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-a50a506e17f747e0a940bc3c982336c6 received from {username='slave'} at 127.0.0.1:32824
I20260202 15:07:53.734091 13487 tablet_copy_client.cc:323] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:53.734318 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-a50a506e17f747e0a940bc3c982336c6 on tablet a50a506e17f747e0a940bc3c982336c6 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:53.735522 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 65f098e59c0240a3a65cbd15dc358b64 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:53.736001 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 65f098e59c0240a3a65cbd15dc358b64 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-65f098e59c0240a3a65cbd15dc358b64
I20260202 15:07:53.742307 12736 tablet_copy_source_session.cc:215] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:53.744546 13487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f098e59c0240a3a65cbd15dc358b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:53.756071 13487 tablet_copy_client.cc:806] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:53.756492 13487 tablet_copy_client.cc:670] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:53.759873 13487 tablet_copy_client.cc:538] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:53.766804 13487 tablet_bootstrap.cc:492] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:53.784060 13487 tablet_bootstrap.cc:492] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:53.784916 13487 tablet_bootstrap.cc:492] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:53.785497 13487 ts_tablet_manager.cc:1403] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.019s user 0.020s sys 0.000s
I20260202 15:07:53.788182 13487 raft_consensus.cc:359] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.788914 13487 raft_consensus.cc:740] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:53.789448 13487 consensus_queue.cc:260] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.791599 13487 ts_tablet_manager.cc:1434] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:07:53.792963 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-65f098e59c0240a3a65cbd15dc358b64 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:53.793351 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-65f098e59c0240a3a65cbd15dc358b64 on tablet 65f098e59c0240a3a65cbd15dc358b64 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:53.826021 13487 ts_tablet_manager.cc:933] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:53.827410 13487 tablet_copy_client.cc:323] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:53.828662 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 00b1a15d2d1246e1abe52586ddf512ef from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:53.829097 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 00b1a15d2d1246e1abe52586ddf512ef from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-00b1a15d2d1246e1abe52586ddf512ef
I20260202 15:07:53.834221 12736 tablet_copy_source_session.cc:215] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:53.836467 13487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b1a15d2d1246e1abe52586ddf512ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:53.841771 13483 ts_tablet_manager.cc:933] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:07:53.850091 13487 tablet_copy_client.cc:806] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:53.850531 13487 tablet_copy_client.cc:670] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:53.851775 13483 tablet_copy_client.cc:323] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:07:53.853153 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 0f70fb0b2ffc4a51b06fe26a22355100 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:07:53.853591 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 0f70fb0b2ffc4a51b06fe26a22355100 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-0f70fb0b2ffc4a51b06fe26a22355100
I20260202 15:07:53.859275 12660 tablet_copy_source_session.cc:215] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:53.859683 13487 tablet_copy_client.cc:538] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:53.862337 13483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f70fb0b2ffc4a51b06fe26a22355100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:53.869743 13487 tablet_bootstrap.cc:492] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:53.875344 13483 tablet_copy_client.cc:806] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:53.875896 13483 tablet_copy_client.cc:670] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:53.879073 13483 tablet_copy_client.cc:538] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:53.885545 13483 tablet_bootstrap.cc:492] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:53.888525 13487 tablet_bootstrap.cc:492] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:53.889251 13487 tablet_bootstrap.cc:492] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:53.889751 13487 ts_tablet_manager.cc:1403] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20260202 15:07:53.892311 13487 raft_consensus.cc:359] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.892968 13487 raft_consensus.cc:740] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:53.893534 13487 consensus_queue.cc:260] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.895422 13487 ts_tablet_manager.cc:1434] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260202 15:07:53.897185 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-00b1a15d2d1246e1abe52586ddf512ef received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:53.897505 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-00b1a15d2d1246e1abe52586ddf512ef on tablet 00b1a15d2d1246e1abe52586ddf512ef with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:53.903009 13483 tablet_bootstrap.cc:492] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:53.903750 13483 tablet_bootstrap.cc:492] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:53.905402 13483 ts_tablet_manager.cc:1403] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20260202 15:07:53.907577 13483 raft_consensus.cc:359] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.908232 13483 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:53.908718 13483 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:53.912005 13483 ts_tablet_manager.cc:1434] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:07:53.915158 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-0f70fb0b2ffc4a51b06fe26a22355100 received from {username='slave'} at 127.0.0.1:42782
I20260202 15:07:53.915500 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-0f70fb0b2ffc4a51b06fe26a22355100 on tablet 0f70fb0b2ffc4a51b06fe26a22355100 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:54.070785 13363 raft_consensus.cc:1217] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:54.127647 13363 raft_consensus.cc:1217] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:54.234910 13363 raft_consensus.cc:1217] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:54.436288 13363 raft_consensus.cc:1217] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:54.627518 13457 raft_consensus.cc:1064] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:54.629521 13457 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:54.634402 12718 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.634657 13363 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.635732 12645 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.635861 13457 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.636835 13496 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.637470 13497 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.650053 12645 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.648974 13497 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.650542 12718 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.666527 12546 catalog_manager.cc:5654] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.668780 13258 raft_consensus.cc:1064] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:54.671248 13258 consensus_queue.cc:237] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:54.668231 13363 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.692674 12799 raft_consensus.cc:1275] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.693943 13363 raft_consensus.cc:1275] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.694013 13495 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:07:54.695712 13495 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.701373 12650 raft_consensus.cc:1275] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.706578 13495 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.720948 12795 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:54.752708 13470 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.759099 13464 raft_consensus.cc:1064] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:54.758399 12650 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.761416 13362 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.765964 12796 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.780525 12546 catalog_manager.cc:5654] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.793311 13464 consensus_queue.cc:237] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:54.795845 13363 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:54.795996 12726 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:54.797240 13505 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:54.798024 13502 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:54.804836 13496 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.806000 13363 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.809388 12722 raft_consensus.cc:2955] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.816138 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet a50a506e17f747e0a940bc3c982336c6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:54.821044 12718 consensus_queue.cc:237] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:54.826102 12795 raft_consensus.cc:1275] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:54.826501 12547 catalog_manager.cc:5654] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:07:54.827322 13362 raft_consensus.cc:1275] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:54.827692 13470 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:54.829506 13470 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:54.835740 12795 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.837437 13261 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.838059 12718 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.839990 13261 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.839800 13258 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.842319 13362 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.843458 13261 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.843212 12799 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.844429 13362 raft_consensus.cc:2955] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.845659 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet a50a506e17f747e0a940bc3c982336c6 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:07:54.864346 12547 catalog_manager.cc:5654] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.865310 13514 tablet_replica.cc:333] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:07:54.869957 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 65f098e59c0240a3a65cbd15dc358b64 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:54.870671 13514 raft_consensus.cc:2243] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:54.871415 13514 raft_consensus.cc:2272] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:54.873771 13494 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.874763 13361 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.883725 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet 65f098e59c0240a3a65cbd15dc358b64 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:07:54.886765 12547 catalog_manager.cc:5654] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.897974 12795 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.904422 13514 ts_tablet_manager.cc:1916] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:54.905452 12718 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.933687 12650 consensus_queue.cc:237] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:54.934944 13514 ts_tablet_manager.cc:1929] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:54.935453 13514 log.cc:1199] T a50a506e17f747e0a940bc3c982336c6 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/a50a506e17f747e0a940bc3c982336c6
I20260202 15:07:54.936898 13514 tablet_replica.cc:333] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:07:54.937387 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet a50a506e17f747e0a940bc3c982336c6 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:54.937973 13514 raft_consensus.cc:2243] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:54.938062 12795 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:54.938633 13514 raft_consensus.cc:2272] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:54.939086 13512 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:54.939246 13362 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:54.940846 13512 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:54.941542 13514 ts_tablet_manager.cc:1916] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:54.946182 13464 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.948508 13362 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.953668 13517 raft_consensus.cc:1064] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:54.955556 13514 ts_tablet_manager.cc:1929] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:54.955888 13514 log.cc:1199] T 65f098e59c0240a3a65cbd15dc358b64 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/65f098e59c0240a3a65cbd15dc358b64
I20260202 15:07:54.955870 13517 consensus_queue.cc:237] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:54.957122 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet 65f098e59c0240a3a65cbd15dc358b64 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:54.960114 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 0f70fb0b2ffc4a51b06fe26a22355100 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:54.962615 12545 catalog_manager.cc:5654] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.964175 12796 raft_consensus.cc:2955] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:54.963858 12795 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.967298 12645 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.968519 13495 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.969419 13516 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:54.972474 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f70fb0b2ffc4a51b06fe26a22355100 with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:07:54.984812 13362 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:54.989266 13495 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:55.011795 13521 tablet_replica.cc:333] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:07:55.012785 13521 raft_consensus.cc:2243] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:55.013697 13521 raft_consensus.cc:2272] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:55.014564 13517 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:55.015054 12645 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:55.017838 13521 ts_tablet_manager.cc:1916] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:55.027129 13362 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:55.033070 12799 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:55.042315 12545 catalog_manager.cc:5654] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:55.063757 13521 ts_tablet_manager.cc:1929] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:55.064153 13521 log.cc:1199] T 0f70fb0b2ffc4a51b06fe26a22355100 P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/0f70fb0b2ffc4a51b06fe26a22355100
I20260202 15:07:55.064488 12718 consensus_queue.cc:237] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:55.066501 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet 0f70fb0b2ffc4a51b06fe26a22355100 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:55.070171 12645 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:55.071491 13517 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:55.075489 13362 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:55.078403 13470 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:55.079221 12645 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:55.082244 13495 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:55.092392 12547 catalog_manager.cc:5654] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:55.094532 13362 raft_consensus.cc:2955] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:55.093807 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 00b1a15d2d1246e1abe52586ddf512ef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:55.107774 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 00b1a15d2d1246e1abe52586ddf512ef with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:07:55.109364 13525 tablet_replica.cc:333] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:07:55.110183 13525 raft_consensus.cc:2243] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:55.110805 13525 raft_consensus.cc:2272] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:55.113493 13525 ts_tablet_manager.cc:1916] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:55.124271 13525 ts_tablet_manager.cc:1929] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:55.124529 13525 log.cc:1199] T 00b1a15d2d1246e1abe52586ddf512ef P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/00b1a15d2d1246e1abe52586ddf512ef
I20260202 15:07:55.125609 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 00b1a15d2d1246e1abe52586ddf512ef (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:56.167105 12722 consensus_queue.cc:237] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.172585 12799 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:07:56.173065 12689 consensus_peers.cc:597] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11a10c0c64914444a6659e661207c082. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:56.173648 12645 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:56.173936 13516 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:56.174724 13470 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:56.181003 13516 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.182674 12648 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.187867 12795 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.191900 12799 consensus_queue.cc:237] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.196187 12544 catalog_manager.cc:5654] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260202 15:07:56.208338 12764 consensus_peers.cc:597] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee143496d46d4a45a2d24f0990e74f44. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:56.209067 12648 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:56.210299 13457 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:56.210959 12722 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:56.212201 13457 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:56.217537 13457 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.223425 12648 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.228267 12722 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.242187 12718 consensus_queue.cc:237] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.244059 12546 catalog_manager.cc:5654] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
W20260202 15:07:56.256143 12689 consensus_peers.cc:597] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ed5f5e04d064d6e8d431aa2209cc141. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:56.256974 12796 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:56.258780 12645 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:56.260035 13517 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:56.261134 13527 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:56.268208 13470 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.271153 12796 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.274003 12648 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.289189 12718 consensus_queue.cc:237] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.290028 12546 catalog_manager.cc:5654] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.308838 12796 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:56.309633 12648 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:56.309809 13543 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:07:56.310657 12689 consensus_peers.cc:597] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 488d33b86df249748c59fbf945f4e8b3. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:56.310858 13470 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:56.318792 12648 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.317246 13543 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.333074 12796 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.341449 12546 catalog_manager.cc:5654] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:56.634621 13547 ts_tablet_manager.cc:933] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:56.636134 13547 tablet_copy_client.cc:323] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:56.637449 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 11a10c0c64914444a6659e661207c082 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:56.637930 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 11a10c0c64914444a6659e661207c082 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-11a10c0c64914444a6659e661207c082
I20260202 15:07:56.645116 12736 tablet_copy_source_session.cc:215] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:56.648401 13547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11a10c0c64914444a6659e661207c082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:56.658636 13547 tablet_copy_client.cc:806] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:56.660778 13547 tablet_copy_client.cc:670] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:56.666618 13547 tablet_copy_client.cc:538] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:56.671693 13549 ts_tablet_manager.cc:933] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:56.672910 13547 tablet_bootstrap.cc:492] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:56.679430 13549 tablet_copy_client.cc:323] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:56.682107 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 4ed5f5e04d064d6e8d431aa2209cc141 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:56.682610 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 4ed5f5e04d064d6e8d431aa2209cc141 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-4ed5f5e04d064d6e8d431aa2209cc141
I20260202 15:07:56.688016 12736 tablet_copy_source_session.cc:215] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:56.690629 13549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed5f5e04d064d6e8d431aa2209cc141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:56.691264 13552 ts_tablet_manager.cc:933] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:07:56.701169 13547 tablet_bootstrap.cc:492] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:56.704447 13547 tablet_bootstrap.cc:492] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:56.705374 13547 ts_tablet_manager.cc:1403] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.003s
I20260202 15:07:56.707508 13547 raft_consensus.cc:359] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.708370 13552 tablet_copy_client.cc:323] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:07:56.708317 13549 tablet_copy_client.cc:806] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:56.708463 13547 raft_consensus.cc:740] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:56.709131 13549 tablet_copy_client.cc:670] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:56.709895 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet ee143496d46d4a45a2d24f0990e74f44 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:07:56.709301 13547 consensus_queue.cc:260] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.710608 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet ee143496d46d4a45a2d24f0990e74f44 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-ee143496d46d4a45a2d24f0990e74f44
I20260202 15:07:56.713261 13547 ts_tablet_manager.cc:1434] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260202 15:07:56.716636 12810 tablet_copy_source_session.cc:215] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:56.719211 13552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee143496d46d4a45a2d24f0990e74f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:56.722934 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-11a10c0c64914444a6659e661207c082 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:56.723344 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-11a10c0c64914444a6659e661207c082 on tablet 11a10c0c64914444a6659e661207c082 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:56.733011 13549 tablet_copy_client.cc:538] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:56.737548 13552 tablet_copy_client.cc:806] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:56.738204 13552 tablet_copy_client.cc:670] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:56.739694 13549 tablet_bootstrap.cc:492] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:56.746402 13552 tablet_copy_client.cc:538] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:56.762933 13552 tablet_bootstrap.cc:492] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:56.764505 13549 tablet_bootstrap.cc:492] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:56.765537 13549 tablet_bootstrap.cc:492] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:56.766088 13549 ts_tablet_manager.cc:1403] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.004s
I20260202 15:07:56.768741 13549 raft_consensus.cc:359] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.769559 13549 raft_consensus.cc:740] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:56.770119 13549 consensus_queue.cc:260] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.772243 13549 ts_tablet_manager.cc:1434] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260202 15:07:56.773949 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-4ed5f5e04d064d6e8d431aa2209cc141 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:56.774359 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-4ed5f5e04d064d6e8d431aa2209cc141 on tablet 4ed5f5e04d064d6e8d431aa2209cc141 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:56.774988 13547 ts_tablet_manager.cc:933] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:56.778288 13547 tablet_copy_client.cc:323] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:56.780129 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 488d33b86df249748c59fbf945f4e8b3 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:56.780599 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 488d33b86df249748c59fbf945f4e8b3 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-488d33b86df249748c59fbf945f4e8b3
I20260202 15:07:56.786583 12736 tablet_copy_source_session.cc:215] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:56.789211 13547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 488d33b86df249748c59fbf945f4e8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:56.792572 13552 tablet_bootstrap.cc:492] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:56.793339 13552 tablet_bootstrap.cc:492] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:56.793841 13552 ts_tablet_manager.cc:1403] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.001s
I20260202 15:07:56.796331 13552 raft_consensus.cc:359] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.797082 13552 raft_consensus.cc:740] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:56.797632 13552 consensus_queue.cc:260] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.800292 13552 ts_tablet_manager.cc:1434] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:07:56.804162 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-ee143496d46d4a45a2d24f0990e74f44 received from {username='slave'} at 127.0.0.1:32824
I20260202 15:07:56.804507 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-ee143496d46d4a45a2d24f0990e74f44 on tablet ee143496d46d4a45a2d24f0990e74f44 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:56.813616 13547 tablet_copy_client.cc:806] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:56.814038 13547 tablet_copy_client.cc:670] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:56.816509 13547 tablet_copy_client.cc:538] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:56.823663 13547 tablet_bootstrap.cc:492] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:56.838351 13547 tablet_bootstrap.cc:492] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:56.838984 13547 tablet_bootstrap.cc:492] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:56.839478 13547 ts_tablet_manager.cc:1403] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260202 15:07:56.841435 13547 raft_consensus.cc:359] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.842187 13547 raft_consensus.cc:740] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:56.842829 13547 consensus_queue.cc:260] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:56.844857 13547 ts_tablet_manager.cc:1434] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:07:56.846277 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-488d33b86df249748c59fbf945f4e8b3 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:56.846714 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-488d33b86df249748c59fbf945f4e8b3 on tablet 488d33b86df249748c59fbf945f4e8b3 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:57.073926 13363 raft_consensus.cc:1217] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:57.157806 13363 raft_consensus.cc:1217] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:57.160018 13363 raft_consensus.cc:1217] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:57.337666 13363 raft_consensus.cc:1217] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:07:57.524765 13560 raft_consensus.cc:1064] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:57.526747 13560 consensus_queue.cc:237] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:57.531451 12796 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.531807 13363 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.532042 12650 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.532758 13516 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.533674 13559 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.534340 13470 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.552850 13559 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.553933 12650 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.567625 13363 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.568423 12544 catalog_manager.cc:5654] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.570787 12799 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.600836 12726 consensus_queue.cc:237] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:57.605279 13363 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:57.607262 13559 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:57.608518 12650 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:57.609761 13559 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:57.616389 13516 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.619069 13363 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.619267 12645 raft_consensus.cc:2955] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.626844 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4ed5f5e04d064d6e8d431aa2209cc141 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:57.629371 13470 raft_consensus.cc:1064] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:57.631435 13470 consensus_queue.cc:237] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:57.635200 12545 catalog_manager.cc:5654] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.638895 12645 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.638187 13363 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.640157 12799 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.641230 13560 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.642091 13470 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.642699 13559 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.645287 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed5f5e04d064d6e8d431aa2209cc141 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:07:57.649600 13572 tablet_replica.cc:333] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:07:57.650966 13572 raft_consensus.cc:2243] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:57.651917 13572 raft_consensus.cc:2272] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:57.661128 13572 ts_tablet_manager.cc:1916] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:57.676702 13470 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.678360 13362 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.685793 13572 ts_tablet_manager.cc:1929] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:57.686209 13538 raft_consensus.cc:1064] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:57.686247 13572 log.cc:1199] T 4ed5f5e04d064d6e8d431aa2209cc141 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/4ed5f5e04d064d6e8d431aa2209cc141
I20260202 15:07:57.688202 13538 consensus_queue.cc:237] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:57.702239 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 4ed5f5e04d064d6e8d431aa2209cc141 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:57.699601 12645 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.704469 12544 catalog_manager.cc:5654] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.722258 12799 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.727914 13362 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.729912 12726 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.730453 13457 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.731887 13457 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260202 15:07:57.736614 12645 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.742229 13576 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.742877 12722 consensus_queue.cc:237] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:57.755442 12650 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:57.757177 13559 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:57.767807 13560 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.779708 12648 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.786603 13361 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:57.791456 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 11a10c0c64914444a6659e661207c082 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:57.796386 13560 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:57.796079 13470 raft_consensus.cc:1064] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:07:57.799551 13470 consensus_queue.cc:237] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:57.804153 12544 catalog_manager.cc:5654] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.808835 13361 raft_consensus.cc:2955] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.817059 12795 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.818439 13560 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.821516 13363 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.824857 12648 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.826108 13470 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:07:57.831650 13363 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:07:57.818091 13576 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.838232 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 11a10c0c64914444a6659e661207c082 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:07:57.835662 12546 catalog_manager.cc:5654] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.839148 13572 tablet_replica.cc:333] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:07:57.840714 13572 raft_consensus.cc:2243] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:57.841264 13572 raft_consensus.cc:2272] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:57.843263 13560 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:07:57.845578 13572 ts_tablet_manager.cc:1916] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:57.855525 12726 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.870318 12645 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.879446 13560 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.886802 12796 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.895859 12795 consensus_queue.cc:237] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:57.898332 12644 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.900537 13572 ts_tablet_manager.cc:1929] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:57.901000 13572 log.cc:1199] T 11a10c0c64914444a6659e661207c082 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/11a10c0c64914444a6659e661207c082
I20260202 15:07:57.900990 12546 catalog_manager.cc:5654] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:57.904733 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 11a10c0c64914444a6659e661207c082 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:57.904985 13361 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.907732 13363 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:57.910337 13576 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:57.917308 12645 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:57.920120 13457 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:57.921379 13538 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.928125 12645 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.929231 13363 raft_consensus.cc:2955] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.945176 12547 catalog_manager.cc:5654] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.945874 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet ee143496d46d4a45a2d24f0990e74f44 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:57.947499 12718 consensus_queue.cc:237] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:07:57.952903 13362 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:57.953691 12795 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:07:57.954084 13560 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:07:57.956810 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet ee143496d46d4a45a2d24f0990e74f44 with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:07:57.957703 13470 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260202 15:07:57.964217 13470 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.968151 13362 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.973240 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 488d33b86df249748c59fbf945f4e8b3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:07:57.974319 12798 raft_consensus.cc:2955] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:07:57.991504 12545 catalog_manager.cc:5654] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:07:57.997260 13586 tablet_replica.cc:333] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:07:57.998148 13586 raft_consensus.cc:2243] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:57.998943 13586 raft_consensus.cc:2272] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:58.001945 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet 488d33b86df249748c59fbf945f4e8b3 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:07:58.008193 13586 ts_tablet_manager.cc:1916] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:58.015799 13587 tablet_replica.cc:333] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:07:58.016764 13587 raft_consensus.cc:2243] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:07:58.017455 13587 raft_consensus.cc:2272] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:07:58.022176 13587 ts_tablet_manager.cc:1916] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:07:58.026188 13586 ts_tablet_manager.cc:1929] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:58.026654 13586 log.cc:1199] T ee143496d46d4a45a2d24f0990e74f44 P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/ee143496d46d4a45a2d24f0990e74f44
I20260202 15:07:58.028160 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet ee143496d46d4a45a2d24f0990e74f44 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:58.034410 13587 ts_tablet_manager.cc:1929] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:07:58.034729 13587 log.cc:1199] T 488d33b86df249748c59fbf945f4e8b3 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/488d33b86df249748c59fbf945f4e8b3
I20260202 15:07:58.035851 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet 488d33b86df249748c59fbf945f4e8b3 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:07:59.053072 12718 consensus_queue.cc:237] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.059334 12799 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:59.059813 12645 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:07:59.061106 12689 consensus_peers.cc:597] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9592591cd602455aa52fdc1d84e48b63. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:59.061615 13560 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:07:59.062778 13559 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:59.070890 13559 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.071835 12645 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.079998 12799 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.094399 12545 catalog_manager.cc:5654] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:59.100010 12718 consensus_queue.cc:237] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.110581 12798 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260202 15:07:59.111549 12689 consensus_peers.cc:597] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc5a4c8371434c48926f011a134d3bd4. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:59.112186 12644 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:07:59.113348 13591 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:07:59.114327 13594 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:07:59.125397 13594 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.139065 12799 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.144613 12645 consensus_queue.cc:237] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.148252 12648 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.161339 12547 catalog_manager.cc:5654] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
W20260202 15:07:59.170732 12614 consensus_peers.cc:597] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8f84948a93c4dc984e69bfd1f44be23. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:59.173765 12796 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:59.174150 12718 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:59.175112 13464 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:59.176023 13562 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:59.182438 13464 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.184567 12718 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.187513 12796 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.194360 12726 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.198160 12546 catalog_manager.cc:5654] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:07:59.218788 12799 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:59.220041 12645 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:07:59.219822 13594 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:07:59.221010 12689 consensus_peers.cc:597] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 323d926990494ad1aafb6aa0282d9f68. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:07:59.221037 13560 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:07:59.227154 13591 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.228503 12645 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.228643 12799 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.241140 12547 catalog_manager.cc:5654] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:07:59.501235 13609 ts_tablet_manager.cc:933] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:59.504315 13609 tablet_copy_client.cc:323] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:59.505774 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 9592591cd602455aa52fdc1d84e48b63 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:59.506214 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 9592591cd602455aa52fdc1d84e48b63 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-9592591cd602455aa52fdc1d84e48b63
I20260202 15:07:59.514333 13611 ts_tablet_manager.cc:933] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:59.515388 12736 tablet_copy_source_session.cc:215] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:59.516592 13611 tablet_copy_client.cc:323] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:59.518179 13609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9592591cd602455aa52fdc1d84e48b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:59.518187 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet dc5a4c8371434c48926f011a134d3bd4 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:59.518860 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet dc5a4c8371434c48926f011a134d3bd4 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-dc5a4c8371434c48926f011a134d3bd4
I20260202 15:07:59.524627 12736 tablet_copy_source_session.cc:215] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:59.527760 13611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc5a4c8371434c48926f011a134d3bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:59.532279 13609 tablet_copy_client.cc:806] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:59.532960 13609 tablet_copy_client.cc:670] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:59.536629 13609 tablet_copy_client.cc:538] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:59.541096 13611 tablet_copy_client.cc:806] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:59.541683 13611 tablet_copy_client.cc:670] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:59.544675 13609 tablet_bootstrap.cc:492] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:59.547591 13611 tablet_copy_client.cc:538] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:59.556509 13611 tablet_bootstrap.cc:492] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:59.568674 13609 tablet_bootstrap.cc:492] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:59.569410 13609 tablet_bootstrap.cc:492] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:59.569882 13609 ts_tablet_manager.cc:1403] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.008s
I20260202 15:07:59.572358 13609 raft_consensus.cc:359] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.573173 13609 raft_consensus.cc:740] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:59.573799 13609 consensus_queue.cc:260] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.581362 13611 tablet_bootstrap.cc:492] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:59.583010 13611 tablet_bootstrap.cc:492] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:59.583580 13611 ts_tablet_manager.cc:1403] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.013s
I20260202 15:07:59.585877 13611 raft_consensus.cc:359] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.586403 13611 raft_consensus.cc:740] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:59.587726 13611 consensus_queue.cc:260] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.590610 13611 ts_tablet_manager.cc:1434] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260202 15:07:59.593315 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-dc5a4c8371434c48926f011a134d3bd4 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:59.593679 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-dc5a4c8371434c48926f011a134d3bd4 on tablet dc5a4c8371434c48926f011a134d3bd4 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:59.594523 13609 ts_tablet_manager.cc:1434] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.024s user 0.011s sys 0.008s
I20260202 15:07:59.597765 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-9592591cd602455aa52fdc1d84e48b63 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:59.598111 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-9592591cd602455aa52fdc1d84e48b63 on tablet 9592591cd602455aa52fdc1d84e48b63 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:59.609100 13609 ts_tablet_manager.cc:933] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:07:59.610436 13609 tablet_copy_client.cc:323] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:07:59.611676 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 323d926990494ad1aafb6aa0282d9f68 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:07:59.612064 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 323d926990494ad1aafb6aa0282d9f68 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-323d926990494ad1aafb6aa0282d9f68
I20260202 15:07:59.617913 12736 tablet_copy_source_session.cc:215] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:59.621816 13609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323d926990494ad1aafb6aa0282d9f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:59.637409 13609 tablet_copy_client.cc:806] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:59.637830 13609 tablet_copy_client.cc:670] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:59.643666 13609 tablet_copy_client.cc:538] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:59.650353 13609 tablet_bootstrap.cc:492] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:59.666899 13609 tablet_bootstrap.cc:492] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:59.667601 13609 tablet_bootstrap.cc:492] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:59.668105 13609 ts_tablet_manager.cc:1403] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.005s
I20260202 15:07:59.670558 13609 raft_consensus.cc:359] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.671258 13609 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:59.671844 13609 consensus_queue.cc:260] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.674026 13609 ts_tablet_manager.cc:1434] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260202 15:07:59.678040 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-323d926990494ad1aafb6aa0282d9f68 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:07:59.679699 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-323d926990494ad1aafb6aa0282d9f68 on tablet 323d926990494ad1aafb6aa0282d9f68 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:59.742252 13609 ts_tablet_manager.cc:933] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:07:59.743708 13609 tablet_copy_client.cc:323] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:07:59.745180 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet f8f84948a93c4dc984e69bfd1f44be23 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:07:59.745571 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet f8f84948a93c4dc984e69bfd1f44be23 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-f8f84948a93c4dc984e69bfd1f44be23
I20260202 15:07:59.753789 12660 tablet_copy_source_session.cc:215] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:07:59.756534 13609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f84948a93c4dc984e69bfd1f44be23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:07:59.769402 13609 tablet_copy_client.cc:806] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:07:59.769758 13609 tablet_copy_client.cc:670] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:07:59.774569 13609 tablet_copy_client.cc:538] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:07:59.779820 13609 tablet_bootstrap.cc:492] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:07:59.795764 13609 tablet_bootstrap.cc:492] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:07:59.796418 13609 tablet_bootstrap.cc:492] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:07:59.796864 13609 ts_tablet_manager.cc:1403] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260202 15:07:59.799909 13609 raft_consensus.cc:359] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.800577 13609 raft_consensus.cc:740] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:07:59.801141 13609 consensus_queue.cc:260] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:07:59.803159 13609 ts_tablet_manager.cc:1434] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260202 15:07:59.804936 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-f8f84948a93c4dc984e69bfd1f44be23 received from {username='slave'} at 127.0.0.1:42782
I20260202 15:07:59.805352 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-f8f84948a93c4dc984e69bfd1f44be23 on tablet f8f84948a93c4dc984e69bfd1f44be23 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:07:59.997651 13363 raft_consensus.cc:1217] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260202 15:08:00.088382 13363 raft_consensus.cc:1217] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:00.141412 13363 raft_consensus.cc:1217] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:00.215468 13363 raft_consensus.cc:1217] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:00.377794 13560 raft_consensus.cc:1064] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:00.379525 13560 consensus_queue.cc:237] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:00.384760 12796 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260202 15:08:00.386467 13594 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.386641 13362 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260202 15:08:00.386601 12648 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260202 15:08:00.388906 13594 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.389582 13560 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.400242 13594 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.401405 12644 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.408345 13362 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.417954 12545 catalog_manager.cc:5654] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:00.433038 12799 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.470743 12718 consensus_queue.cc:237] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:00.474812 13362 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260202 15:08:00.475952 12644 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260202 15:08:00.479398 13627 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260202 15:08:00.481973 13627 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:00.486773 13591 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.495573 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet dc5a4c8371434c48926f011a134d3bd4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:00.495414 13362 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.495852 12644 raft_consensus.cc:2955] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.507186 12547 catalog_manager.cc:5654] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.516013 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet dc5a4c8371434c48926f011a134d3bd4 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:08:00.518553 13632 tablet_replica.cc:333] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:08:00.520948 13632 raft_consensus.cc:2243] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:00.523204 13632 raft_consensus.cc:2272] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:00.527925 13632 ts_tablet_manager.cc:1916] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:00.539319 13632 ts_tablet_manager.cc:1929] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260202 15:08:00.539624 13632 log.cc:1199] T dc5a4c8371434c48926f011a134d3bd4 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/dc5a4c8371434c48926f011a134d3bd4
I20260202 15:08:00.540927 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet dc5a4c8371434c48926f011a134d3bd4 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:00.629623 13621 raft_consensus.cc:1064] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:00.631721 13621 consensus_queue.cc:237] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:00.636824 13362 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:00.637866 13621 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.638286 12798 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:00.638777 12726 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:00.640338 13603 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.640934 13464 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.665012 13621 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.667542 12718 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.669704 13362 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.680472 12798 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.689051 12546 catalog_manager.cc:5654] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:00.704193 12645 consensus_queue.cc:237] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:00.706743 13594 raft_consensus.cc:1064] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:00.708842 13594 consensus_queue.cc:237] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:00.716670 12798 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:00.717834 13464 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:00.723852 12799 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:00.724085 13362 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:00.724114 13361 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:00.725378 13464 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:00.725695 13591 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.727169 13591 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.728380 12648 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:00.735556 13560 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.735821 13603 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.737105 13362 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.744915 12799 raft_consensus.cc:2955] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.746961 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet f8f84948a93c4dc984e69bfd1f44be23 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:00.753322 12544 catalog_manager.cc:5654] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:00.756187 13627 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.757583 12795 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.764377 13560 raft_consensus.cc:1064] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:00.767129 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet f8f84948a93c4dc984e69bfd1f44be23 with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:08:00.766163 13560 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:00.780575 13643 tablet_replica.cc:333] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:08:00.781503 13643 raft_consensus.cc:2243] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:00.782265 13643 raft_consensus.cc:2272] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:00.786167 13643 ts_tablet_manager.cc:1916] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:00.797185 12644 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.798703 12545 catalog_manager.cc:5654] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:00.795248 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 52449.1us
I20260202 15:08:00.803974 12795 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:00.807909 13362 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:00.808218 13646 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.808897 12645 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:00.814554 13591 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.816069 13591 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:00.835268 13643 ts_tablet_manager.cc:1929] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:00.835737 13643 log.cc:1199] T f8f84948a93c4dc984e69bfd1f44be23 P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/f8f84948a93c4dc984e69bfd1f44be23
I20260202 15:08:00.837296 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet f8f84948a93c4dc984e69bfd1f44be23 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:00.847204 13594 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.847929 12799 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.848562 12645 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.856024 13362 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.864739 13363 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.871752 12726 consensus_queue.cc:237] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:00.878115 12544 catalog_manager.cc:5654] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.890971 12799 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:00.892881 13591 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:00.894899 13361 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:00.899420 13591 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:00.902715 13627 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.908823 12722 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:00.910857 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 9592591cd602455aa52fdc1d84e48b63 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:00.913645 13361 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.914124 12799 raft_consensus.cc:2955] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.915781 12645 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:00.916591 13362 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:00.920460 13594 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:00.921963 13591 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:00.923995 12547 catalog_manager.cc:5654] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:00.929769 13594 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.937053 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet 9592591cd602455aa52fdc1d84e48b63 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:08:00.938488 13361 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.941126 12645 raft_consensus.cc:2955] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.945645 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 323d926990494ad1aafb6aa0282d9f68 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:00.950541 12547 catalog_manager.cc:5654] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:00.957403 13652 tablet_replica.cc:333] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:08:00.959048 13652 raft_consensus.cc:2243] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:00.960469 13652 raft_consensus.cc:2272] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:00.965749 13652 ts_tablet_manager.cc:1916] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:00.966127 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 323d926990494ad1aafb6aa0282d9f68 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:08:00.967011 13632 tablet_replica.cc:333] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:08:00.967926 13632 raft_consensus.cc:2243] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:00.968477 13632 raft_consensus.cc:2272] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:00.971894 13632 ts_tablet_manager.cc:1916] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:00.977209 13652 ts_tablet_manager.cc:1929] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:00.977502 13652 log.cc:1199] T 9592591cd602455aa52fdc1d84e48b63 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/9592591cd602455aa52fdc1d84e48b63
I20260202 15:08:00.978845 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet 9592591cd602455aa52fdc1d84e48b63 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:00.983055 13632 ts_tablet_manager.cc:1929] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:00.983348 13632 log.cc:1199] T 323d926990494ad1aafb6aa0282d9f68 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/323d926990494ad1aafb6aa0282d9f68
I20260202 15:08:00.984288 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 323d926990494ad1aafb6aa0282d9f68 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:02.026487 12795 consensus_queue.cc:237] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.033041 12718 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:02.033468 12645 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:02.033511 12764 consensus_peers.cc:597] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4687c40305f64dd2960fcbb336f93395. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:02.034448 13576 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:02.035316 13538 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:02.042518 13656 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.043676 12718 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.043761 12645 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.059253 12545 catalog_manager.cc:5654] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.068989 12795 consensus_queue.cc:237] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.086824 12718 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:02.088133 13656 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:08:02.088908 12764 consensus_peers.cc:597] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d91ac50607b34cd1b7d61cd42f7c26d8. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:02.089394 12645 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:02.090797 13656 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:02.098235 13576 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.104697 12718 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.108215 12645 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.120301 12648 consensus_queue.cc:237] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.123646 12546 catalog_manager.cc:5654] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.136255 12718 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:02.136735 12614 consensus_peers.cc:597] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4d8302d577042feaa44f63d596dc87d. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:02.137346 12796 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:02.140578 13644 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:08:02.141665 13603 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:02.148622 13644 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.152043 12726 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.178360 12545 catalog_manager.cc:5654] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.184479 12726 consensus_queue.cc:237] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.188450 12795 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.197111 12796 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:02.198673 13594 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:08:02.203118 12689 consensus_peers.cc:597] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d30f8f27ce00483c85fb1758a4f81f03. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:02.203896 12648 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:02.205125 13627 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:02.209276 12796 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.207540 13627 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.220726 12648 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:02.229956 12544 catalog_manager.cc:5654] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:02.485248 13673 ts_tablet_manager.cc:933] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:02.487323 13673 tablet_copy_client.cc:323] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:08:02.488816 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet d91ac50607b34cd1b7d61cd42f7c26d8 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:08:02.489259 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet d91ac50607b34cd1b7d61cd42f7c26d8 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-d91ac50607b34cd1b7d61cd42f7c26d8
I20260202 15:08:02.494288 12810 tablet_copy_source_session.cc:215] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:02.497475 13673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91ac50607b34cd1b7d61cd42f7c26d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:02.510502 13673 tablet_copy_client.cc:806] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:02.510917 13673 tablet_copy_client.cc:670] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:02.514240 13673 tablet_copy_client.cc:538] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:02.520012 13673 tablet_bootstrap.cc:492] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:02.542822 13673 tablet_bootstrap.cc:492] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:02.543453 13673 tablet_bootstrap.cc:492] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:02.543926 13673 ts_tablet_manager.cc:1403] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.005s
I20260202 15:08:02.546113 13673 raft_consensus.cc:359] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.546830 13673 raft_consensus.cc:740] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:02.547394 13673 consensus_queue.cc:260] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.572167 13673 ts_tablet_manager.cc:1434] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.028s user 0.009s sys 0.018s
I20260202 15:08:02.573550 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-d91ac50607b34cd1b7d61cd42f7c26d8 received from {username='slave'} at 127.0.0.1:32824
I20260202 15:08:02.573906 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-d91ac50607b34cd1b7d61cd42f7c26d8 on tablet d91ac50607b34cd1b7d61cd42f7c26d8 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:02.582845 13673 ts_tablet_manager.cc:933] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:02.584123 13673 tablet_copy_client.cc:323] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:08:02.585228 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet 4687c40305f64dd2960fcbb336f93395 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:08:02.585724 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet 4687c40305f64dd2960fcbb336f93395 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-4687c40305f64dd2960fcbb336f93395
I20260202 15:08:02.590785 12810 tablet_copy_source_session.cc:215] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:02.593062 13673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4687c40305f64dd2960fcbb336f93395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:02.605475 13673 tablet_copy_client.cc:806] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:02.605932 13673 tablet_copy_client.cc:670] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:02.610322 13673 tablet_copy_client.cc:538] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:02.611357 13678 ts_tablet_manager.cc:933] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:02.618451 13678 tablet_copy_client.cc:323] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:02.619367 13673 tablet_bootstrap.cc:492] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:02.620256 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet e4d8302d577042feaa44f63d596dc87d from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:02.620699 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet e4d8302d577042feaa44f63d596dc87d from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-e4d8302d577042feaa44f63d596dc87d
I20260202 15:08:02.627846 12660 tablet_copy_source_session.cc:215] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:02.630499 13678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d8302d577042feaa44f63d596dc87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:02.637933 13673 tablet_bootstrap.cc:492] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:02.638657 13673 tablet_bootstrap.cc:492] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:02.639206 13673 ts_tablet_manager.cc:1403] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.001s
I20260202 15:08:02.641501 13673 raft_consensus.cc:359] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.642302 13673 raft_consensus.cc:740] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:02.643019 13673 consensus_queue.cc:260] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.645221 13673 ts_tablet_manager.cc:1434] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:08:02.645256 13678 tablet_copy_client.cc:806] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:02.646095 13678 tablet_copy_client.cc:670] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:02.647156 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-4687c40305f64dd2960fcbb336f93395 received from {username='slave'} at 127.0.0.1:32824
I20260202 15:08:02.647504 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-4687c40305f64dd2960fcbb336f93395 on tablet 4687c40305f64dd2960fcbb336f93395 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:02.650539 13678 tablet_copy_client.cc:538] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:02.659260 13678 tablet_bootstrap.cc:492] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:02.672641 13678 tablet_bootstrap.cc:492] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:02.673368 13678 tablet_bootstrap.cc:492] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:02.673897 13678 ts_tablet_manager.cc:1403] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260202 15:08:02.677400 13678 raft_consensus.cc:359] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.678258 13678 raft_consensus.cc:740] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:02.678872 13678 consensus_queue.cc:260] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.680980 13678 ts_tablet_manager.cc:1434] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260202 15:08:02.682582 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-e4d8302d577042feaa44f63d596dc87d received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:02.683043 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-e4d8302d577042feaa44f63d596dc87d on tablet e4d8302d577042feaa44f63d596dc87d with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:02.743458 13678 ts_tablet_manager.cc:933] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:02.744846 13678 tablet_copy_client.cc:323] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:08:02.746001 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet d30f8f27ce00483c85fb1758a4f81f03 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:08:02.746433 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet d30f8f27ce00483c85fb1758a4f81f03 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-d30f8f27ce00483c85fb1758a4f81f03
I20260202 15:08:02.751700 12736 tablet_copy_source_session.cc:215] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:02.754298 13678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30f8f27ce00483c85fb1758a4f81f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:02.768306 13678 tablet_copy_client.cc:806] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:02.768769 13678 tablet_copy_client.cc:670] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:02.772701 13678 tablet_copy_client.cc:538] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:02.779084 13678 tablet_bootstrap.cc:492] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:02.796182 13678 tablet_bootstrap.cc:492] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:02.796842 13678 tablet_bootstrap.cc:492] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:02.797266 13678 ts_tablet_manager.cc:1403] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20260202 15:08:02.799250 13678 raft_consensus.cc:359] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.799749 13678 raft_consensus.cc:740] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:02.800231 13678 consensus_queue.cc:260] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:02.802255 13678 ts_tablet_manager.cc:1434] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:08:02.804160 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-d30f8f27ce00483c85fb1758a4f81f03 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:08:02.804457 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-d30f8f27ce00483c85fb1758a4f81f03 on tablet d30f8f27ce00483c85fb1758a4f81f03 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:02.974318 13361 raft_consensus.cc:1217] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:03.037799 13361 raft_consensus.cc:1217] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:03.089938 13361 raft_consensus.cc:1217] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:03.157260 13361 raft_consensus.cc:1217] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:03.360183 13538 raft_consensus.cc:1064] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:03.362075 13538 consensus_queue.cc:237] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:03.367010 12718 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.367630 13361 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.369097 12645 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.373085 13686 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.375442 13686 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.377725 13689 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.388720 13656 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.391346 12718 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.395323 12645 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.406960 12546 catalog_manager.cc:5654] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.416505 13361 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.435905 12799 consensus_queue.cc:237] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:03.440483 12718 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:03.442800 13361 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:03.444521 13686 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:03.445194 13688 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:03.450484 13686 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.451838 12718 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.460880 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet d91ac50607b34cd1b7d61cd42f7c26d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:03.461145 13361 raft_consensus.cc:2955] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.463642 12544 catalog_manager.cc:5654] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:03.478576 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet d91ac50607b34cd1b7d61cd42f7c26d8 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:08:03.481971 13667 raft_consensus.cc:1064] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:03.485816 13667 consensus_queue.cc:237] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:03.499279 13697 tablet_replica.cc:333] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:08:03.500387 13697 raft_consensus.cc:2243] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:03.500890 13697 raft_consensus.cc:2272] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:03.500980 13361 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.502424 13685 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.505214 13697 ts_tablet_manager.cc:1916] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:03.507046 12799 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.509570 12718 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.509821 13667 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.511370 13669 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.534798 13697 ts_tablet_manager.cc:1929] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:03.535637 13697 log.cc:1199] T d91ac50607b34cd1b7d61cd42f7c26d8 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/d91ac50607b34cd1b7d61cd42f7c26d8
I20260202 15:08:03.536810 13667 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.537976 12718 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.547482 13361 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.552534 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet d91ac50607b34cd1b7d61cd42f7c26d8 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:03.555239 12544 catalog_manager.cc:5654] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.559427 12796 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.575706 13688 raft_consensus.cc:1064] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:03.577493 13688 consensus_queue.cc:237] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:03.586186 12726 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.587282 13688 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.587502 12644 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.588639 13688 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.591311 13361 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.592711 13688 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:08:03.601562 12645 consensus_queue.cc:237] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:03.601688 13538 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.606304 13362 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:03.606351 12799 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:03.607625 13667 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:03.609014 13667 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:03.612046 12645 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.618397 13667 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.618824 12718 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.616082 13362 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.621229 12547 catalog_manager.cc:5654] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:03.637705 13361 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.640831 12796 raft_consensus.cc:2955] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.652527 12799 consensus_queue.cc:237] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:03.654160 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet e4d8302d577042feaa44f63d596dc87d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:03.656800 12547 catalog_manager.cc:5654] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.678205 13627 raft_consensus.cc:1064] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:03.680961 13627 consensus_queue.cc:237] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:03.684259 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet e4d8302d577042feaa44f63d596dc87d with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:08:03.691375 12798 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.693684 13363 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.695712 13360 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:03.695894 13708 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.696545 12648 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:03.699057 13656 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:03.702024 13684 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260202 15:08:03.706171 13538 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:03.709368 13538 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.711229 12648 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.719128 13361 raft_consensus.cc:2955] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.721550 12645 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:03.724694 13684 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:03.732463 13710 tablet_replica.cc:333] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:08:03.738137 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4687c40305f64dd2960fcbb336f93395 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:03.739813 13710 raft_consensus.cc:2243] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:03.740609 13710 raft_consensus.cc:2272] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:03.752941 12547 catalog_manager.cc:5654] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:03.767277 13710 ts_tablet_manager.cc:1916] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:03.770931 13594 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.768472 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet 4687c40305f64dd2960fcbb336f93395 with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:08:03.775657 12798 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.786221 12645 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.787503 13362 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.801540 13710 ts_tablet_manager.cc:1929] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:03.801971 13710 log.cc:1199] T e4d8302d577042feaa44f63d596dc87d P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/e4d8302d577042feaa44f63d596dc87d
I20260202 15:08:03.803498 13710 tablet_replica.cc:333] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:08:03.804387 13710 raft_consensus.cc:2243] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:03.804589 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet e4d8302d577042feaa44f63d596dc87d (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:03.805068 13710 raft_consensus.cc:2272] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:03.809090 12544 catalog_manager.cc:5654] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.809435 13710 ts_tablet_manager.cc:1916] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:03.828805 13710 ts_tablet_manager.cc:1929] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:03.829169 13710 log.cc:1199] T 4687c40305f64dd2960fcbb336f93395 P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/4687c40305f64dd2960fcbb336f93395
I20260202 15:08:03.831458 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet 4687c40305f64dd2960fcbb336f93395 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:03.833917 12722 consensus_queue.cc:237] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:03.839394 13362 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:03.839883 12798 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:03.840350 13684 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260202 15:08:03.842221 13708 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:03.848599 13711 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.849869 12798 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.852016 13362 raft_consensus.cc:2955] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.858219 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet d30f8f27ce00483c85fb1758a4f81f03 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:03.862659 12545 catalog_manager.cc:5654] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:03.870286 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet d30f8f27ce00483c85fb1758a4f81f03 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:08:03.871198 13697 tablet_replica.cc:333] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:08:03.871841 13697 raft_consensus.cc:2243] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:03.872386 13697 raft_consensus.cc:2272] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:03.875277 13697 ts_tablet_manager.cc:1916] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:03.885466 13697 ts_tablet_manager.cc:1929] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:03.885731 13697 log.cc:1199] T d30f8f27ce00483c85fb1758a4f81f03 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/d30f8f27ce00483c85fb1758a4f81f03
I20260202 15:08:03.886848 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet d30f8f27ce00483c85fb1758a4f81f03 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:04.921387 12648 consensus_queue.cc:237] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:08:04.926328 12614 consensus_peers.cc:597] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3a3797f335d43dca87344d8954a1158. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:04.931818 12796 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:04.931990 12718 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:04.933296 13667 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:04.933981 13706 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:04.941203 13667 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:04.942575 12796 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:04.944033 12718 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:04.952144 12648 consensus_queue.cc:237] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:04.957144 12545 catalog_manager.cc:5654] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
W20260202 15:08:04.968998 12614 consensus_peers.cc:597] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d28e4999b504df4a53610187731416d. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:04.974361 12799 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:04.975437 12722 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:04.976501 13706 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:04.977617 13667 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:04.989331 13718 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:04.991321 12796 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:05.001349 12718 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:05.004236 12796 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.010907 12648 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:05.013437 12764 consensus_peers.cc:597] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2476071e1d24aa6a1342c8d81699cef. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:05.016857 13686 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:05.022089 12718 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:05.023418 13538 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:05.024488 12544 catalog_manager.cc:5654] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:05.024434 13730 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:05.026836 12648 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:05.039228 12718 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:05.048532 12796 consensus_queue.cc:237] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.051352 12547 catalog_manager.cc:5654] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:05.069069 12718 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:05.070623 12648 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:05.071722 13730 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:05.073626 13538 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:08:05.074260 12764 consensus_peers.cc:597] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f9321e2251943229a0b7dbdc0577c6b. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:05.081809 13686 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:05.083670 12648 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:05.097994 12718 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:05.108389 12545 catalog_manager.cc:5654] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:05.350193 13737 ts_tablet_manager.cc:933] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:05.351840 13737 tablet_copy_client.cc:323] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:05.353425 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 7d28e4999b504df4a53610187731416d from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:05.353845 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 7d28e4999b504df4a53610187731416d from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-7d28e4999b504df4a53610187731416d
I20260202 15:08:05.359879 12660 tablet_copy_source_session.cc:215] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:05.362401 13737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d28e4999b504df4a53610187731416d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:05.374320 13737 tablet_copy_client.cc:806] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:05.374821 13737 tablet_copy_client.cc:670] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:05.378690 13737 tablet_copy_client.cc:538] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:05.385120 13737 tablet_bootstrap.cc:492] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:05.403170 13737 tablet_bootstrap.cc:492] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:05.403888 13737 tablet_bootstrap.cc:492] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:05.404350 13737 ts_tablet_manager.cc:1403] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.001s
I20260202 15:08:05.406489 13737 raft_consensus.cc:359] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.407171 13737 raft_consensus.cc:740] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:05.407685 13737 consensus_queue.cc:260] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.411312 13737 ts_tablet_manager.cc:1434] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260202 15:08:05.412683 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-7d28e4999b504df4a53610187731416d received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:05.413089 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-7d28e4999b504df4a53610187731416d on tablet 7d28e4999b504df4a53610187731416d with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:05.429451 13737 ts_tablet_manager.cc:933] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:05.431205 13737 tablet_copy_client.cc:323] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:05.432565 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet d3a3797f335d43dca87344d8954a1158 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:05.433032 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet d3a3797f335d43dca87344d8954a1158 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-d3a3797f335d43dca87344d8954a1158
I20260202 15:08:05.438218 12660 tablet_copy_source_session.cc:215] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:05.441277 13737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a3797f335d43dca87344d8954a1158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:05.453131 13737 tablet_copy_client.cc:806] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:05.453656 13737 tablet_copy_client.cc:670] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:05.460142 13742 ts_tablet_manager.cc:933] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:05.461879 13737 tablet_copy_client.cc:538] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:05.462185 13742 tablet_copy_client.cc:323] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:08:05.463959 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet d2476071e1d24aa6a1342c8d81699cef from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:08:05.464357 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet d2476071e1d24aa6a1342c8d81699cef from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-d2476071e1d24aa6a1342c8d81699cef
I20260202 15:08:05.469218 13737 tablet_bootstrap.cc:492] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:05.469830 12810 tablet_copy_source_session.cc:215] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:05.472245 13742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2476071e1d24aa6a1342c8d81699cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:05.487471 13737 tablet_bootstrap.cc:492] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:05.488308 13737 tablet_bootstrap.cc:492] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:05.488835 13742 tablet_copy_client.cc:806] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:05.488893 13737 ts_tablet_manager.cc:1403] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20260202 15:08:05.489439 13742 tablet_copy_client.cc:670] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:05.491586 13737 raft_consensus.cc:359] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.492347 13737 raft_consensus.cc:740] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:05.494520 13737 consensus_queue.cc:260] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.496662 13737 ts_tablet_manager.cc:1434] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260202 15:08:05.498289 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-d3a3797f335d43dca87344d8954a1158 received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:05.498629 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-d3a3797f335d43dca87344d8954a1158 on tablet d3a3797f335d43dca87344d8954a1158 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:05.502071 13742 tablet_copy_client.cc:538] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:05.508603 13742 tablet_bootstrap.cc:492] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:05.525714 13742 tablet_bootstrap.cc:492] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:05.526417 13742 tablet_bootstrap.cc:492] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:05.526960 13742 ts_tablet_manager.cc:1403] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.007s
I20260202 15:08:05.529073 13742 raft_consensus.cc:359] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.529592 13742 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:05.530015 13742 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.531972 13742 ts_tablet_manager.cc:1434] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:08:05.533720 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-d2476071e1d24aa6a1342c8d81699cef received from {username='slave'} at 127.0.0.1:32824
I20260202 15:08:05.534065 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-d2476071e1d24aa6a1342c8d81699cef on tablet d2476071e1d24aa6a1342c8d81699cef with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:05.567950 13742 ts_tablet_manager.cc:933] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:05.570267 13742 tablet_copy_client.cc:323] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:08:05.571373 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet 6f9321e2251943229a0b7dbdc0577c6b from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:08:05.571731 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet 6f9321e2251943229a0b7dbdc0577c6b from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-6f9321e2251943229a0b7dbdc0577c6b
I20260202 15:08:05.584702 12810 tablet_copy_source_session.cc:215] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:05.587237 13742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9321e2251943229a0b7dbdc0577c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:05.600862 13742 tablet_copy_client.cc:806] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:05.601279 13742 tablet_copy_client.cc:670] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:05.604189 13742 tablet_copy_client.cc:538] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:05.610103 13742 tablet_bootstrap.cc:492] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:05.625814 13742 tablet_bootstrap.cc:492] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:05.626497 13742 tablet_bootstrap.cc:492] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:05.627023 13742 ts_tablet_manager.cc:1403] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.017s user 0.019s sys 0.000s
I20260202 15:08:05.629236 13742 raft_consensus.cc:359] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.629956 13742 raft_consensus.cc:740] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:05.630611 13742 consensus_queue.cc:260] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:05.632867 13742 ts_tablet_manager.cc:1434] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:08:05.634454 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-6f9321e2251943229a0b7dbdc0577c6b received from {username='slave'} at 127.0.0.1:32824
I20260202 15:08:05.634819 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-6f9321e2251943229a0b7dbdc0577c6b on tablet 6f9321e2251943229a0b7dbdc0577c6b with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:05.807226 13361 raft_consensus.cc:1217] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:05.855867 13361 raft_consensus.cc:1217] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:06.031733 13362 raft_consensus.cc:1217] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:06.052703 13362 raft_consensus.cc:1217] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:06.335134 13718 raft_consensus.cc:1064] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:06.337294 13718 consensus_queue.cc:237] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:06.352787 12799 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.352820 13362 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.354009 13706 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.354749 13721 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.355716 12726 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.357757 13721 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.379841 13721 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.381013 12722 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.391407 13362 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.395200 12799 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.395524 12547 catalog_manager.cc:5654] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:06.427417 13736 raft_consensus.cc:1064] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:06.429742 13730 raft_consensus.cc:1064] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:06.429533 13736 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:06.431833 13730 consensus_queue.cc:237] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:06.432739 12650 consensus_queue.cc:237] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:06.437791 12718 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:06.437866 13362 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:06.439147 13721 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:06.443208 13749 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:06.452790 13362 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.459331 13718 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.460508 12718 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.462002 13757 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.464533 12718 raft_consensus.cc:2955] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.470211 13721 raft_consensus.cc:1064] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:06.470118 12545 catalog_manager.cc:5654] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.472313 13721 consensus_queue.cc:237] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:06.477465 13362 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.478638 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 7d28e4999b504df4a53610187731416d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:06.478853 12648 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.480077 13757 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.480996 13730 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.487277 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d28e4999b504df4a53610187731416d with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:08:06.490478 13763 tablet_replica.cc:333] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:08:06.491456 13763 raft_consensus.cc:2243] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:06.492147 13763 raft_consensus.cc:2272] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:06.496088 13763 ts_tablet_manager.cc:1916] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:06.497967 13757 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.501663 12722 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.502373 13360 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.502849 13686 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.504560 13686 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.499588 12648 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.509145 12650 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.512354 13730 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.526398 12545 catalog_manager.cc:5654] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:06.533819 13763 ts_tablet_manager.cc:1929] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:06.534274 13763 log.cc:1199] T 7d28e4999b504df4a53610187731416d P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/7d28e4999b504df4a53610187731416d
I20260202 15:08:06.533514 12718 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.536810 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 7d28e4999b504df4a53610187731416d (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:06.548890 13736 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.557433 12650 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.559934 13360 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.564028 13361 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.567323 13363 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.568006 12795 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.568375 12718 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:06.569110 13749 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.569811 13718 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.572122 13718 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:06.588356 12722 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.594295 13706 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.597970 12718 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.598235 12545 catalog_manager.cc:5654] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.604522 12795 consensus_queue.cc:237] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:06.615160 13361 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:06.615381 12726 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:06.616510 12795 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.618068 13757 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:06.618970 13730 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:06.633213 13686 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.633693 13360 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.643625 12718 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.645812 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 6f9321e2251943229a0b7dbdc0577c6b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:06.649899 12795 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:06.652508 13362 raft_consensus.cc:2955] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.658624 12650 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:06.660038 13736 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260202 15:08:06.660634 13363 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:06.661171 12545 catalog_manager.cc:5654] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.667124 13757 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:06.669513 12546 catalog_manager.cc:5654] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.675498 12650 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.674427 13736 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.686797 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet d2476071e1d24aa6a1342c8d81699cef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:06.687516 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f9321e2251943229a0b7dbdc0577c6b with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:08:06.686878 13362 raft_consensus.cc:2955] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.695202 13772 tablet_replica.cc:333] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:08:06.696112 13772 raft_consensus.cc:2243] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:06.696722 13772 raft_consensus.cc:2272] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:06.700388 13772 ts_tablet_manager.cc:1916] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:06.712742 12545 catalog_manager.cc:5654] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.712926 12644 consensus_queue.cc:237] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:06.719053 13362 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:06.721312 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet d2476071e1d24aa6a1342c8d81699cef with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:08:06.725746 12718 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:06.727701 13706 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:06.728796 13773 tablet_replica.cc:333] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:08:06.729755 13773 raft_consensus.cc:2243] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:06.730405 13773 raft_consensus.cc:2272] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:06.735045 13749 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.736382 13772 ts_tablet_manager.cc:1929] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:06.736774 13772 log.cc:1199] T 6f9321e2251943229a0b7dbdc0577c6b P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/6f9321e2251943229a0b7dbdc0577c6b
I20260202 15:08:06.738893 13773 ts_tablet_manager.cc:1916] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:06.739333 13361 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.745565 13721 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:06.748886 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet 6f9321e2251943229a0b7dbdc0577c6b (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:06.749397 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet d3a3797f335d43dca87344d8954a1158 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:06.749732 12718 raft_consensus.cc:2955] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.750851 12545 catalog_manager.cc:5654] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:06.762032 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet d3a3797f335d43dca87344d8954a1158 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:08:06.762965 13763 tablet_replica.cc:333] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:08:06.764000 13763 raft_consensus.cc:2243] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:06.764556 13763 raft_consensus.cc:2272] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:06.768312 13763 ts_tablet_manager.cc:1916] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:06.776029 13773 ts_tablet_manager.cc:1929] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:06.776443 13773 log.cc:1199] T d2476071e1d24aa6a1342c8d81699cef P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/d2476071e1d24aa6a1342c8d81699cef
I20260202 15:08:06.778445 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet d2476071e1d24aa6a1342c8d81699cef (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:06.785132 13763 ts_tablet_manager.cc:1929] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:06.785465 13763 log.cc:1199] T d3a3797f335d43dca87344d8954a1158 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/d3a3797f335d43dca87344d8954a1158
I20260202 15:08:06.786417 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet d3a3797f335d43dca87344d8954a1158 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:07.813472 12799 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:07.820147 12718 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:07.821370 13736 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:07.824095 12650 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:07.823416 12764 consensus_peers.cc:597] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b57a31e08c1420ab954d13d66af9297. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:07.825551 13757 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:07.830684 13736 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.833220 12650 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.835135 12718 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.844333 12545 catalog_manager.cc:5654] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:07.854591 12796 consensus_queue.cc:237] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:08:07.862990 12764 consensus_peers.cc:597] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3e391e37e424b1b8b2c63352ec8c198. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:07.864404 12718 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:07.864355 12650 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:07.865994 13736 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:07.874627 13757 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:08:07.878115 13778 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.883956 12650 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.884521 12718 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.893553 12644 consensus_queue.cc:237] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:07.896771 12544 catalog_manager.cc:5654] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:07.902803 12796 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:07.904059 12718 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:07.905261 13776 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:07.906617 13776 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:08:07.909549 12614 consensus_peers.cc:597] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 965bcb05a43147b7be53449d0fe46509. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:07.917783 13775 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.921244 12718 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.922014 12796 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.929540 12644 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:07.933058 12545 catalog_manager.cc:5654] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260202 15:08:07.956692 12614 consensus_peers.cc:597] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d879eebd1924ad8b6f65647f4e73362. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:07.958530 12799 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:07.959628 13706 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:07.960317 12718 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:07.961498 13706 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:07.966470 13775 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.967846 12796 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.972782 12726 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:07.981858 12545 catalog_manager.cc:5654] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:08.399999 13797 ts_tablet_manager.cc:933] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:08.404788 13798 ts_tablet_manager.cc:933] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:08.406399 13797 tablet_copy_client.cc:323] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:08.409104 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 965bcb05a43147b7be53449d0fe46509 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:08.409880 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 965bcb05a43147b7be53449d0fe46509 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-965bcb05a43147b7be53449d0fe46509
I20260202 15:08:08.414325 13798 tablet_copy_client.cc:323] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:08:08.416046 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet 3b57a31e08c1420ab954d13d66af9297 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:08:08.416584 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet 3b57a31e08c1420ab954d13d66af9297 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-3b57a31e08c1420ab954d13d66af9297
I20260202 15:08:08.421198 12660 tablet_copy_source_session.cc:215] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:08.424640 12810 tablet_copy_source_session.cc:215] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:08.424661 13797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965bcb05a43147b7be53449d0fe46509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:08.429493 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b57a31e08c1420ab954d13d66af9297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:08.445456 13798 tablet_copy_client.cc:806] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:08.446015 13798 tablet_copy_client.cc:670] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:08.449406 13798 tablet_copy_client.cc:538] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:08.451859 13797 tablet_copy_client.cc:806] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:08.452368 13797 tablet_copy_client.cc:670] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:08.457139 13797 tablet_copy_client.cc:538] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:08.458175 13798 tablet_bootstrap.cc:492] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:08.480505 13797 tablet_bootstrap.cc:492] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:08.481309 13798 tablet_bootstrap.cc:492] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:08.484169 13798 tablet_bootstrap.cc:492] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:08.484977 13798 ts_tablet_manager.cc:1403] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.008s
I20260202 15:08:08.503183 13804 ts_tablet_manager.cc:933] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:08.506387 13798 raft_consensus.cc:359] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:08.507642 13798 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:08.508448 13798 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:08.511593 13805 ts_tablet_manager.cc:933] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:08.519737 13805 tablet_copy_client.cc:323] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:08.523929 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 4d879eebd1924ad8b6f65647f4e73362 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:08.524446 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 4d879eebd1924ad8b6f65647f4e73362 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-4d879eebd1924ad8b6f65647f4e73362
I20260202 15:08:08.538663 12660 tablet_copy_source_session.cc:215] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:08.522224 13804 tablet_copy_client.cc:323] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:08:08.540951 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet a3e391e37e424b1b8b2c63352ec8c198 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:08:08.541496 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet a3e391e37e424b1b8b2c63352ec8c198 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-a3e391e37e424b1b8b2c63352ec8c198
I20260202 15:08:08.541903 13805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d879eebd1924ad8b6f65647f4e73362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:08.544121 13797 tablet_bootstrap.cc:492] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:08.544907 13797 tablet_bootstrap.cc:492] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:08.545490 13797 ts_tablet_manager.cc:1403] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.065s user 0.017s sys 0.020s
I20260202 15:08:08.547828 13797 raft_consensus.cc:359] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:08.548580 13797 raft_consensus.cc:740] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:08.549183 13797 consensus_queue.cc:260] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:08.548646 12810 tablet_copy_source_session.cc:215] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:08.565443 13804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e391e37e424b1b8b2c63352ec8c198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:08.546654 13798 ts_tablet_manager.cc:1434] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.061s user 0.009s sys 0.004s
I20260202 15:08:08.570152 13797 ts_tablet_manager.cc:1434] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.024s user 0.006s sys 0.012s
I20260202 15:08:08.573153 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-3b57a31e08c1420ab954d13d66af9297 received from {username='slave'} at 127.0.0.1:32824
I20260202 15:08:08.573658 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-3b57a31e08c1420ab954d13d66af9297 on tablet 3b57a31e08c1420ab954d13d66af9297 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:08.573768 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-965bcb05a43147b7be53449d0fe46509 received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:08.574316 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-965bcb05a43147b7be53449d0fe46509 on tablet 965bcb05a43147b7be53449d0fe46509 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:08.576025 13805 tablet_copy_client.cc:806] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:08.579428 13805 tablet_copy_client.cc:670] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:08.587530 13805 tablet_copy_client.cc:538] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:08.589090 13804 tablet_copy_client.cc:806] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:08.589560 13804 tablet_copy_client.cc:670] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:08.596844 13805 tablet_bootstrap.cc:492] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:08.598042 13804 tablet_copy_client.cc:538] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:08.605284 13804 tablet_bootstrap.cc:492] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:08.614444 13805 tablet_bootstrap.cc:492] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:08.615149 13805 tablet_bootstrap.cc:492] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:08.615643 13805 ts_tablet_manager.cc:1403] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.010s
I20260202 15:08:08.617902 13805 raft_consensus.cc:359] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:08.618539 13805 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:08.619092 13805 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:08.629607 13805 ts_tablet_manager.cc:1434] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260202 15:08:08.635784 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-4d879eebd1924ad8b6f65647f4e73362 received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:08.636152 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-4d879eebd1924ad8b6f65647f4e73362 on tablet 4d879eebd1924ad8b6f65647f4e73362 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:08.643021 13804 tablet_bootstrap.cc:492] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:08.643847 13804 tablet_bootstrap.cc:492] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:08.644377 13804 ts_tablet_manager.cc:1403] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.039s user 0.027s sys 0.005s
I20260202 15:08:08.646731 13804 raft_consensus.cc:359] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:08.647446 13804 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:08.647958 13804 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:08.650005 13804 ts_tablet_manager.cc:1434] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:08:08.652329 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-a3e391e37e424b1b8b2c63352ec8c198 received from {username='slave'} at 127.0.0.1:32824
I20260202 15:08:08.652638 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-a3e391e37e424b1b8b2c63352ec8c198 on tablet a3e391e37e424b1b8b2c63352ec8c198 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:08.906279 13361 raft_consensus.cc:1217] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:08.995196 13360 raft_consensus.cc:1217] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:09.008478 13360 raft_consensus.cc:1217] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:09.122536 13360 raft_consensus.cc:1217] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:09.422205 13757 raft_consensus.cc:1064] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:09.424829 13757 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:09.434931 13360 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.435002 12722 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.436573 13778 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.436784 12644 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.438166 13778 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.440595 13736 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.455072 13736 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.456457 12722 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.461046 12645 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.475450 12545 catalog_manager.cc:5654] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.468134 13360 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.487087 13706 raft_consensus.cc:1064] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:09.489118 13706 consensus_queue.cc:237] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:09.508513 13360 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.508656 12795 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.508648 12722 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.510222 13775 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.510989 13706 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.513008 13749 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:08:09.529711 13749 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.535270 12799 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.537772 12795 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:09.546653 12726 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:09.546815 13361 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.549582 13757 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:09.555230 12722 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.557490 13757 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.559893 12726 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.573050 12546 catalog_manager.cc:5654] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:09.579704 13360 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:09.581831 13736 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:09.595022 12645 consensus_queue.cc:237] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:09.598723 13736 raft_consensus.cc:1064] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:09.598451 13360 raft_consensus.cc:2955] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.601050 13736 consensus_queue.cc:237] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:09.603744 13363 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:09.605029 13775 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:09.607168 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 3b57a31e08c1420ab954d13d66af9297 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:09.612409 12722 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:09.614332 13706 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:09.625728 13363 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.624986 13706 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.628186 12645 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.627969 13778 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.634037 13778 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.638928 12726 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.635088 13360 raft_consensus.cc:2955] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.641820 12722 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.643539 13778 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.643903 13813 raft_consensus.cc:1064] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:09.646569 13813 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:09.647645 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 965bcb05a43147b7be53449d0fe46509 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:09.658463 12544 catalog_manager.cc:5654] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.660969 13361 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.665225 12726 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.662923 13757 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.676752 13749 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.678299 13775 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260202 15:08:09.684181 13363 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.667784 12546 catalog_manager.cc:5654] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:09.669340 12798 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:09.670076 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b57a31e08c1420ab954d13d66af9297 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:08:09.705245 13813 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:09.714269 13834 tablet_replica.cc:333] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:08:09.715538 13834 raft_consensus.cc:2243] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:09.716351 13834 raft_consensus.cc:2272] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:09.725533 13834 ts_tablet_manager.cc:1916] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:09.728353 12718 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.730594 12544 catalog_manager.cc:5654] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.747180 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 965bcb05a43147b7be53449d0fe46509 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
W20260202 15:08:09.750032 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 74934.4us
I20260202 15:08:09.761934 12722 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.761230 12650 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.769784 13813 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.760257 13834 ts_tablet_manager.cc:1929] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:09.784657 13834 log.cc:1199] T 3b57a31e08c1420ab954d13d66af9297 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/3b57a31e08c1420ab954d13d66af9297
I20260202 15:08:09.787655 13363 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.786963 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet 3b57a31e08c1420ab954d13d66af9297 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:09.795056 13837 tablet_replica.cc:333] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:08:09.796175 13837 raft_consensus.cc:2243] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:09.796921 13837 raft_consensus.cc:2272] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:09.809414 12547 catalog_manager.cc:5654] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.822645 12796 consensus_queue.cc:237] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:09.818236 13837 ts_tablet_manager.cc:1916] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:09.829300 13363 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:09.831125 12645 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:09.835690 13778 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:09.836804 13832 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:09.840088 12796 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.858304 13837 ts_tablet_manager.cc:1929] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:09.858981 13837 log.cc:1199] T 965bcb05a43147b7be53449d0fe46509 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/965bcb05a43147b7be53449d0fe46509
I20260202 15:08:09.863085 12650 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:09.863193 13832 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.865574 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 965bcb05a43147b7be53449d0fe46509 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:09.869040 13360 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:09.870611 13813 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:09.871711 12798 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:09.873216 12650 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.874788 13361 raft_consensus.cc:2955] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.877507 13749 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:09.883220 13775 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.885653 13360 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.893172 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet a3e391e37e424b1b8b2c63352ec8c198 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:09.894817 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4d879eebd1924ad8b6f65647f4e73362 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:09.894423 12544 catalog_manager.cc:5654] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:09.908667 12796 raft_consensus.cc:2955] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:09.914986 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet a3e391e37e424b1b8b2c63352ec8c198 with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:08:09.919448 13839 tablet_replica.cc:333] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:08:09.921100 13839 raft_consensus.cc:2243] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:09.921746 13839 raft_consensus.cc:2272] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:09.923432 12544 catalog_manager.cc:5654] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:09.929095 13839 ts_tablet_manager.cc:1916] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:09.933710 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d879eebd1924ad8b6f65647f4e73362 with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:08:09.942235 13839 ts_tablet_manager.cc:1929] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:09.942641 13839 log.cc:1199] T a3e391e37e424b1b8b2c63352ec8c198 P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/a3e391e37e424b1b8b2c63352ec8c198
I20260202 15:08:09.944044 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet a3e391e37e424b1b8b2c63352ec8c198 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:09.944180 13839 tablet_replica.cc:333] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:08:09.944881 13839 raft_consensus.cc:2243] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:09.945353 13839 raft_consensus.cc:2272] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:09.947799 13839 ts_tablet_manager.cc:1916] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:09.956125 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:09.957965 13839 ts_tablet_manager.cc:1929] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:09.958317 13839 log.cc:1199] T 4d879eebd1924ad8b6f65647f4e73362 P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/4d879eebd1924ad8b6f65647f4e73362
I20260202 15:08:09.959470 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet 4d879eebd1924ad8b6f65647f4e73362 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:09.995784 12718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:09.996202 12718 raft_consensus.cc:606] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:10.007920 12645 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:10.008371 12645 raft_consensus.cc:606] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:10.010311 12718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:10.010669 12718 raft_consensus.cc:606] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:10.028364 12796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:10.028745 12796 raft_consensus.cc:606] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:10.030515 12718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:10.030871 12718 raft_consensus.cc:606] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:10.032547 12796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:10.032968 12796 raft_consensus.cc:606] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:10.033870 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260202 15:08:10.035017 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:10.035784 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:10.041949 13838 raft_consensus.cc:993] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:10.042627 13767 raft_consensus.cc:1081] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:10.044478 12798 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56734
I20260202 15:08:10.045007 12798 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:10.045365 12798 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.051371 12798 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:10.053520 12798 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:10.054028 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:10.054414 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:10.054597 12722 raft_consensus.cc:3055] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:10.054884 12645 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.054898 12722 raft_consensus.cc:740] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:10.058423 12645 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:10.058768 12722 consensus_queue.cc:260] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:10.059383 12763 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:10.059945 13778 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:10.060120 12722 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.060356 13778 raft_consensus.cc:697] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:10.061524 13778 consensus_queue.cc:237] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:10.066100 12722 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:10.071496 12544 catalog_manager.cc:5654] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:08:10.104127 13838 raft_consensus.cc:993] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:10.104528 13767 raft_consensus.cc:1081] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:10.105816 13360 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44940
I20260202 15:08:10.106251 13360 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:10.106510 13360 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:10.113128 13360 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.115195 13360 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:10.115458 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:10.115609 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:10.116041 12645 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:10.116103 12718 raft_consensus.cc:3055] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:10.116438 12718 raft_consensus.cc:740] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:10.117040 12718 consensus_queue.cc:260] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.117960 12718 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:10.120941 13838 raft_consensus.cc:993] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:10.122093 13801 raft_consensus.cc:1081] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:10.120961 12645 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:10.123371 13326 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:10.124233 13810 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:10.124577 13360 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44940
I20260202 15:08:10.125020 13360 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:10.125473 13360 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.125593 13810 raft_consensus.cc:697] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:10.126441 13810 consensus_queue.cc:237] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.129225 12718 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:10.131314 13360 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.132690 13360 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:10.133266 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:10.133785 12645 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.133949 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:10.134472 12718 raft_consensus.cc:3055] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:10.134784 12718 raft_consensus.cc:740] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:10.135511 12718 consensus_queue.cc:260] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.136564 12718 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.139377 12645 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:10.140161 13326 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:10.140780 13810 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:10.141191 13810 raft_consensus.cc:697] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:10.142194 12718 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:10.142505 13810 consensus_queue.cc:237] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.144127 12544 catalog_manager.cc:5654] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 2 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 3 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:10.154568 12544 catalog_manager.cc:5654] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:10.381659 13775 raft_consensus.cc:993] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:10.382025 13749 raft_consensus.cc:1081] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:10.383210 12798 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56722
I20260202 15:08:10.383709 12798 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:10.384004 12798 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.387826 12798 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:10.389245 12798 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:10.389786 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:10.389786 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:10.390410 12718 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.390455 12645 raft_consensus.cc:3055] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:10.390779 12645 raft_consensus.cc:740] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:10.391366 12645 consensus_queue.cc:260] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:10.392319 12645 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.395844 12718 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:10.396706 12765 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:10.397235 13757 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:10.397615 13757 raft_consensus.cc:697] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:10.397686 12645 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:10.398262 13757 consensus_queue.cc:237] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:10.406333 12544 catalog_manager.cc:5654] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:08:10.439764 13757 raft_consensus.cc:993] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:10.440158 13778 raft_consensus.cc:1081] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:10.441185 13360 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44934
I20260202 15:08:10.441499 13360 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:10.441737 13360 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.446254 13360 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.447592 13360 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:10.448192 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:10.448355 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:10.448683 12645 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.448786 12796 raft_consensus.cc:3055] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:10.449091 12796 raft_consensus.cc:740] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:10.449728 12796 consensus_queue.cc:260] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.450803 12796 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.454258 12645 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:10.455085 13326 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:10.455691 13810 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:10.456090 13810 raft_consensus.cc:697] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:10.457008 12796 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:10.456761 13810 consensus_queue.cc:237] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.468055 12544 catalog_manager.cc:5654] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:10.470288 13757 raft_consensus.cc:993] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:10.470660 13778 raft_consensus.cc:1081] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:10.471856 13360 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44934
I20260202 15:08:10.472268 13360 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:10.472564 13360 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.480162 13360 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.484766 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:10.485082 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:10.485356 12796 raft_consensus.cc:3055] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:10.485580 12650 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.485921 13360 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:10.485664 12796 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:10.486764 12796 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.487607 12796 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:10.490232 12650 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:10.491004 13326 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:10.491287 12796 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:10.491677 13810 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:10.492300 13810 raft_consensus.cc:697] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:10.496802 13810 consensus_queue.cc:237] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:10.504009 12544 catalog_manager.cc:5654] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:10.507193 12650 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:10.510037 13778 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:10.522486 12718 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:10.523856 13778 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:10.548017 12650 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:10.549077 13810 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:10.560642 12645 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:10.566151 12722 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:10.565260 13857 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:10.567634 13845 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:10.578186 12722 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:10.579663 13845 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260202 15:08:10.841709 12722 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:10.842671 13757 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:10.848593 12650 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:10.849800 13845 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:10.851434 12650 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:10.852525 13778 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:10.867550 12798 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:10.868922 13860 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:10.949319 12650 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:10.950279 13857 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:10.959000 12796 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:10.960776 13810 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:10.961031 12798 consensus_queue.cc:237] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:10.970978 12650 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:10.971117 12722 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:10.971335 12764 consensus_peers.cc:597] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4798df17a1e4d18a594bb1b37b98bae. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:10.972360 13757 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:10.973699 13832 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:10.981067 13757 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:10.981871 12650 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:10.997187 12545 catalog_manager.cc:5654] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.000139 12718 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.005743 12722 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.015787 12798 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:11.019522 13767 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260202 15:08:11.020597 12689 consensus_peers.cc:597] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87df3a9146bb41e1ac7468e17d2d9498. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:11.022027 12650 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:11.024546 13767 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:11.040851 13767 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.045672 12645 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.045651 12796 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.052042 12798 consensus_queue.cc:237] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.061322 12545 catalog_manager.cc:5654] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
W20260202 15:08:11.062079 12764 consensus_peers.cc:597] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ac5476ea30a46548db9ede417d946e6. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:11.063135 12645 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:11.064038 12722 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:11.064374 13757 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:11.070953 13757 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:11.082269 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:11.087422 13757 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.091416 12718 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.096753 12645 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.103863 12545 catalog_manager.cc:5654] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:11.110723 12650 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.123131 12799 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:11.125326 12614 consensus_peers.cc:597] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9335cdc844e44d608b9cdd8369f9afb1. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:11.125926 13775 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:11.126228 12718 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:11.127549 13775 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:11.137282 13829 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.151532 12798 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.154294 12718 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.166559 12545 catalog_manager.cc:5654] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:11.195456 12799 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:11.196038 12799 raft_consensus.cc:606] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:11.198175 12718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:11.199167 12718 raft_consensus.cc:606] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:11.203346 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7d28e4999b504df4a53610187731416d"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:11.203840 12650 raft_consensus.cc:606] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:11.206116 12796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:11.206547 12796 raft_consensus.cc:606] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:11.209554 12718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:11.210062 12718 raft_consensus.cc:606] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:11.211481 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260202 15:08:11.212535 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:11.213136 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:11.255494 13832 raft_consensus.cc:993] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:11.255895 13757 raft_consensus.cc:1081] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:11.257200 13361 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44934
I20260202 15:08:11.257545 13361 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:11.257774 13361 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.262198 13361 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.263847 13361 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:11.264825 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:11.265295 12718 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.268608 12718 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:11.269420 13328 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:11.270073 13810 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:11.270502 13810 raft_consensus.cc:697] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:11.270944 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:11.271427 12799 raft_consensus.cc:3055] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:11.271234 13810 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.271697 12799 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:11.272322 12799 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.273342 12799 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.279990 13775 raft_consensus.cc:993] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:11.280381 13775 raft_consensus.cc:1081] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:11.281800 12544 catalog_manager.cc:5654] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:11.283331 13360 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "7d28e4999b504df4a53610187731416d"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44942
I20260202 15:08:11.283828 13360 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:11.284122 13360 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.284610 12799 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:11.292227 13360 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.293974 13360 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:11.294502 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:11.294664 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:11.296074 12645 raft_consensus.cc:3055] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:11.296141 12726 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.296423 12645 raft_consensus.cc:740] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:11.297031 12645 consensus_queue.cc:260] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.297899 12645 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.302616 12726 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:11.305943 13328 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:11.306676 13860 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:11.307125 13860 raft_consensus.cc:697] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:11.307984 12645 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:11.307874 13860 consensus_queue.cc:237] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.323372 12544 catalog_manager.cc:5654] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:11.448136 13885 ts_tablet_manager.cc:933] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:11.450321 13885 tablet_copy_client.cc:323] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:08:11.451735 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 87df3a9146bb41e1ac7468e17d2d9498 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:08:11.452183 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 87df3a9146bb41e1ac7468e17d2d9498 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-87df3a9146bb41e1ac7468e17d2d9498
I20260202 15:08:11.458230 12736 tablet_copy_source_session.cc:215] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:11.460580 13885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87df3a9146bb41e1ac7468e17d2d9498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:11.472354 13885 tablet_copy_client.cc:806] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:11.474114 13885 tablet_copy_client.cc:670] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:11.477512 13885 tablet_copy_client.cc:538] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:11.484177 13885 tablet_bootstrap.cc:492] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:11.499939 13885 tablet_bootstrap.cc:492] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:11.500923 13885 tablet_bootstrap.cc:492] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:11.501696 13888 ts_tablet_manager.cc:933] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:11.501897 13885 ts_tablet_manager.cc:1403] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260202 15:08:11.504279 13888 tablet_copy_client.cc:323] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:08:11.504317 13885 raft_consensus.cc:359] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.505158 13885 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:11.505791 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet 2ac5476ea30a46548db9ede417d946e6 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:08:11.506203 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet 2ac5476ea30a46548db9ede417d946e6 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-2ac5476ea30a46548db9ede417d946e6
I20260202 15:08:11.510948 13885 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.511428 12810 tablet_copy_source_session.cc:215] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:11.513273 13885 ts_tablet_manager.cc:1434] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.011s user 0.000s sys 0.004s
I20260202 15:08:11.515267 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-87df3a9146bb41e1ac7468e17d2d9498 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:08:11.515614 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-87df3a9146bb41e1ac7468e17d2d9498 on tablet 87df3a9146bb41e1ac7468e17d2d9498 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:11.519387 13885 ts_tablet_manager.cc:933] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:11.526913 13885 tablet_copy_client.cc:323] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:08:11.527143 13888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac5476ea30a46548db9ede417d946e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:11.529021 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet c4798df17a1e4d18a594bb1b37b98bae from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:08:11.529461 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet c4798df17a1e4d18a594bb1b37b98bae from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-c4798df17a1e4d18a594bb1b37b98bae
I20260202 15:08:11.533900 12810 tablet_copy_source_session.cc:215] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:11.536163 13885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4798df17a1e4d18a594bb1b37b98bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:11.546578 13888 tablet_copy_client.cc:806] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:11.547212 13888 tablet_copy_client.cc:670] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:11.549177 13885 tablet_copy_client.cc:806] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:11.549604 13885 tablet_copy_client.cc:670] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:11.554499 13885 tablet_copy_client.cc:538] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:11.556048 13888 tablet_copy_client.cc:538] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:11.564046 13885 tablet_bootstrap.cc:492] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:11.564105 13888 tablet_bootstrap.cc:492] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:11.585755 13888 tablet_bootstrap.cc:492] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:11.586426 13888 tablet_bootstrap.cc:492] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:11.586971 13888 ts_tablet_manager.cc:1403] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.004s
I20260202 15:08:11.587783 13885 tablet_bootstrap.cc:492] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:11.588513 13885 tablet_bootstrap.cc:492] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:11.589013 13885 ts_tablet_manager.cc:1403] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.009s
I20260202 15:08:11.591151 13885 raft_consensus.cc:359] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.591806 13885 raft_consensus.cc:740] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:11.592247 13888 raft_consensus.cc:359] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.592334 13885 consensus_queue.cc:260] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.593027 13888 raft_consensus.cc:740] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:11.593986 13888 consensus_queue.cc:260] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.596252 13888 ts_tablet_manager.cc:1434] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260202 15:08:11.598341 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-2ac5476ea30a46548db9ede417d946e6 received from {username='slave'} at 127.0.0.1:32824
I20260202 15:08:11.598673 13885 ts_tablet_manager.cc:1434] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260202 15:08:11.598722 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-2ac5476ea30a46548db9ede417d946e6 on tablet 2ac5476ea30a46548db9ede417d946e6 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:11.600925 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-c4798df17a1e4d18a594bb1b37b98bae received from {username='slave'} at 127.0.0.1:32824
I20260202 15:08:11.601251 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-c4798df17a1e4d18a594bb1b37b98bae on tablet c4798df17a1e4d18a594bb1b37b98bae with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:11.604952 13885 ts_tablet_manager.cc:933] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:11.619992 13885 tablet_copy_client.cc:323] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:11.621747 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 9335cdc844e44d608b9cdd8369f9afb1 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:11.622207 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 9335cdc844e44d608b9cdd8369f9afb1 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-9335cdc844e44d608b9cdd8369f9afb1
I20260202 15:08:11.628511 12660 tablet_copy_source_session.cc:215] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:11.631209 13885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9335cdc844e44d608b9cdd8369f9afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:11.642022 13885 tablet_copy_client.cc:806] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:11.642334 13885 tablet_copy_client.cc:670] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:11.645967 13885 tablet_copy_client.cc:538] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:11.651813 13885 tablet_bootstrap.cc:492] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:11.667402 13885 tablet_bootstrap.cc:492] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:11.668097 13885 tablet_bootstrap.cc:492] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:11.668596 13885 ts_tablet_manager.cc:1403] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260202 15:08:11.671175 13838 raft_consensus.cc:993] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:11.671581 13880 raft_consensus.cc:1081] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:11.671662 13885 raft_consensus.cc:359] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.672262 13885 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:11.673043 13361 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44940
I20260202 15:08:11.672735 13885 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:11.673570 13361 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:11.673959 13361 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.674834 13885 ts_tablet_manager.cc:1434] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:08:11.676093 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-9335cdc844e44d608b9cdd8369f9afb1 received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:11.676401 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-9335cdc844e44d608b9cdd8369f9afb1 on tablet 9335cdc844e44d608b9cdd8369f9afb1 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:11.679979 13361 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.682345 13361 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:11.682960 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:11.683476 12798 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.684255 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:11.684572 12726 raft_consensus.cc:3055] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:11.684937 12726 raft_consensus.cc:740] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:11.685566 12726 consensus_queue.cc:260] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.686442 12726 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.688637 12798 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:11.689563 13328 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:11.690218 13810 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:11.690567 13810 raft_consensus.cc:697] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:11.691097 12726 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:11.691259 13810 consensus_queue.cc:237] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.701319 13778 raft_consensus.cc:993] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:11.701695 13832 raft_consensus.cc:1081] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:11.705215 13361 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44934
I20260202 15:08:11.705935 13361 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:11.706224 13361 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.708357 12544 catalog_manager.cc:5654] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:11.715699 13838 raft_consensus.cc:993] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:11.716111 13838 raft_consensus.cc:1081] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:11.718843 12799 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56734
I20260202 15:08:11.719256 12799 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:11.719511 12799 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.725410 12799 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:11.725642 13361 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.727790 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:11.728246 13361 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:11.727828 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:11.728328 12726 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.728708 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:11.729259 12718 raft_consensus.cc:3055] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:11.729508 12718 raft_consensus.cc:740] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:11.730078 12718 consensus_queue.cc:260] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:11.730901 12718 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.731364 12799 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:11.731966 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:11.732411 12798 raft_consensus.cc:3055] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:11.728754 12645 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.732645 12798 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:11.735049 12798 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.735989 12798 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:11.736330 12718 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:11.737269 12765 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:11.737974 13832 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:11.738370 13832 raft_consensus.cc:697] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:11.738747 12645 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:11.739117 13832 consensus_queue.cc:237] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:11.744087 12798 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:11.744956 13328 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:11.745553 13810 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:11.745923 13810 raft_consensus.cc:697] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:11.746546 13810 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:11.742118 12726 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:11.750252 12544 catalog_manager.cc:5654] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:08:11.756575 12545 catalog_manager.cc:5654] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:11.778445 12798 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:11.779487 13810 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:11.811846 12726 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:11.813143 13810 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:11.855698 12726 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:11.856777 13810 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:11.868602 12645 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:11.869905 13860 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:11.926903 13361 raft_consensus.cc:1217] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:11.965636 13361 raft_consensus.cc:1217] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:12.004062 13361 raft_consensus.cc:1217] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:12.088126 12718 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:12.089205 13810 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:12.096612 13361 raft_consensus.cc:1217] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:12.104255 12799 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:12.106091 13860 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:12.254173 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:12.256233 12650 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:12.257426 13757 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:12.268081 12718 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:12.270234 13778 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:12.324290 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "015f643434d84eac8a3774b663eb4220"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:12.324822 12650 raft_consensus.cc:606] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:12.327243 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:12.327880 12650 raft_consensus.cc:606] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:12.330096 12796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:12.330580 12796 raft_consensus.cc:606] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:12.333299 12796 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:12.344142 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f72d171c95a149d890432428ecf80739"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:12.334437 13845 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:12.358718 12650 raft_consensus.cc:606] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:12.360220 13757 raft_consensus.cc:1064] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:12.361752 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:12.362098 12650 raft_consensus.cc:606] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:12.363435 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260202 15:08:12.365368 13757 consensus_queue.cc:237] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.364926 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:12.366752 13829 raft_consensus.cc:993] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:12.367093 12722 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:12.367144 13775 raft_consensus.cc:1081] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:12.367893 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:12.368422 13845 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:12.372690 12799 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f72d171c95a149d890432428ecf80739"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56722
I20260202 15:08:12.373203 12799 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:12.373415 12799 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.383716 12799 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:12.386246 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:12.386659 12648 raft_consensus.cc:3055] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:12.386907 12648 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:12.387486 12648 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:12.388397 12648 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.401998 12648 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:12.403733 12763 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:12.404569 13832 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:12.404981 13832 raft_consensus.cc:697] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:12.405689 13832 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:12.409188 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:12.409479 13908 raft_consensus.cc:1064] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:12.409780 12722 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.411880 13908 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.402555 12799 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:12.429276 12722 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:12.439038 13829 raft_consensus.cc:993] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:12.439445 13829 raft_consensus.cc:1081] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:12.446323 13360 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.449913 13917 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.450387 13360 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.450604 12798 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.450773 12726 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.451923 13838 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.453461 13838 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.456115 13917 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.475339 12648 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.477101 13838 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.479164 12648 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.480643 13757 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.486989 12799 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56722
I20260202 15:08:12.488600 12799 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:12.488961 12799 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.495158 12799 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.497336 12799 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:12.497272 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:12.497804 12645 raft_consensus.cc:3055] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:12.498039 12645 raft_consensus.cc:740] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:12.498584 12645 consensus_queue.cc:260] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.499472 12645 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.503901 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:12.504406 13362 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.509565 12645 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:12.509704 13362 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:12.510637 12763 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:12.511263 13757 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:12.511663 13757 raft_consensus.cc:697] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:12.512387 13757 consensus_queue.cc:237] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.515087 13778 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.529359 13360 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.533164 13775 raft_consensus.cc:1064] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:12.532166 13838 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.535167 13775 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.538885 12547 catalog_manager.cc:5654] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:08:12.542007 12796 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.544564 13362 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.552085 13361 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.552673 12545 catalog_manager.cc:5654] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:12.553354 13913 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.546275 12648 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.568084 12725 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.571810 13778 raft_consensus.cc:1064] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:12.567190 12650 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.580523 12544 catalog_manager.cc:5654] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.589146 12718 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.593075 13778 consensus_queue.cc:237] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.597077 12798 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.598639 13757 raft_consensus.cc:993] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:12.603178 13757 raft_consensus.cc:1081] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:12.610363 13361 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44934
I20260202 15:08:12.610841 13361 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:12.611198 13361 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.618858 13775 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.629280 13829 raft_consensus.cc:993] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:12.629647 13829 raft_consensus.cc:1081] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:12.630950 13360 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44942
I20260202 15:08:12.631420 13360 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:12.631788 13360 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.626426 13935 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260202 15:08:12.637938 13362 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.638522 12648 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.637977 12725 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:12.639539 13360 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.639770 13778 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.641327 13360 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:12.641604 13832 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:12.642231 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:12.647464 12795 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.648329 13935 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.621201 13361 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.652138 13361 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:12.652498 12722 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.655164 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:12.655622 12718 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.655867 12795 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:12.655944 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:12.658835 12796 raft_consensus.cc:3055] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:12.669510 12796 raft_consensus.cc:740] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:12.669739 12645 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:12.658315 13328 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:12.670549 13362 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:12.670832 13860 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:12.671262 13860 raft_consensus.cc:697] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:12.671993 13860 consensus_queue.cc:237] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.672322 12796 consensus_queue.cc:260] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.673391 12796 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.674805 13908 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:12.644281 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:12.681999 12648 raft_consensus.cc:3055] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:12.682264 12648 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:12.683565 12648 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.685001 12648 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.686645 13908 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:12.692061 13940 raft_consensus.cc:993] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:12.691166 13838 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.664721 12795 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.697101 12641 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.666929 13360 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.656394 12798 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:12.692433 13940 raft_consensus.cc:1081] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:12.705746 12721 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "015f643434d84eac8a3774b663eb4220"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:37626
I20260202 15:08:12.705580 13362 raft_consensus.cc:2955] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.706843 12796 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:12.706203 12721 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:12.708232 13328 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:12.708477 12721 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.708979 13810 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:12.709411 13810 raft_consensus.cc:697] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:12.710111 13810 consensus_queue.cc:237] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.650359 13928 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:08:12.661903 12718 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:12.730907 12648 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:12.745412 12545 catalog_manager.cc:5654] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:12.759145 12721 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:12.769798 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:12.770431 12796 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.773401 12721 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:12.746251 12544 catalog_manager.cc:5654] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260202 15:08:12.809832 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 84185.2us
I20260202 15:08:12.810949 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 87df3a9146bb41e1ac7468e17d2d9498 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:12.811218 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:12.811738 12648 raft_consensus.cc:3055] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:12.812006 12648 raft_consensus.cc:740] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:12.812554 12648 consensus_queue.cc:260] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:12.813680 12648 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:12.825825 12796 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:12.828138 12725 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:12.829229 12690 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:12.829159 12544 catalog_manager.cc:5654] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.830385 13946 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:12.830880 13946 raft_consensus.cc:697] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:12.831615 13946 consensus_queue.cc:237] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:12.832584 13917 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:12.830343 12648 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:12.839376 12722 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.840610 13832 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.847985 12718 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:12.858623 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 120408us
I20260202 15:08:12.872784 12547 catalog_manager.cc:5654] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:12.883502 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 87df3a9146bb41e1ac7468e17d2d9498 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:08:12.887303 12648 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.887322 13361 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.919668 12546 catalog_manager.cc:5654] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:12.920361 12641 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.924787 13362 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:12.927501 12795 consensus_queue.cc:237] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:12.928941 12648 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:12.931347 13832 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:12.933085 13361 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:12.933439 13832 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:12.934823 12718 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:12.935458 13362 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:12.936028 13832 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:12.941085 13775 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:12.941622 13757 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.945042 12718 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:12.948167 13775 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:12.954960 13917 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:12.964974 13775 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.938524 12545 catalog_manager.cc:5654] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:12.969559 13360 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.966424 13362 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:12.971798 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet c4798df17a1e4d18a594bb1b37b98bae with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:12.977391 12718 raft_consensus.cc:2955] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:13.016968 12725 raft_consensus.cc:2955] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:13.038745 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 50652.9us
I20260202 15:08:13.069375 13954 tablet_replica.cc:333] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:08:13.070711 13954 raft_consensus.cc:2243] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:13.071591 13954 raft_consensus.cc:2272] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:13.071208 12546 catalog_manager.cc:5654] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:13.086892 12795 consensus_queue.cc:237] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:13.104594 12796 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:13.105845 13845 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260202 15:08:13.118088 13360 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:13.139619 12545 catalog_manager.cc:5654] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:13.143461 13930 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260202 15:08:13.158033 12650 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:13.179237 13757 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:13.186647 12544 catalog_manager.cc:5654] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:13.196709 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 9335cdc844e44d608b9cdd8369f9afb1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:13.195116 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 9335cdc844e44d608b9cdd8369f9afb1 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:08:13.219395 13929 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260202 15:08:13.288856 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 2ac5476ea30a46548db9ede417d946e6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:13.300295 13954 ts_tablet_manager.cc:1916] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:13.321534 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet c4798df17a1e4d18a594bb1b37b98bae with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:08:13.333868 12648 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:13.347096 12648 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:13.352337 13946 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:13.362481 13949 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:13.381397 13970 tablet_replica.cc:333] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:08:13.383837 12644 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:13.386849 12718 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:13.394197 13810 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:13.400673 12645 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:13.413414 12796 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:13.423779 13970 raft_consensus.cc:2243] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:13.439940 13970 raft_consensus.cc:2272] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:13.455502 13360 raft_consensus.cc:2955] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:13.466130 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:13.468128 13965 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260202 15:08:13.482019 13810 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:13.485973 13970 ts_tablet_manager.cc:1916] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:13.503353 13954 ts_tablet_manager.cc:1929] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:13.503777 13954 log.cc:1199] T 87df3a9146bb41e1ac7468e17d2d9498 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/87df3a9146bb41e1ac7468e17d2d9498
I20260202 15:08:13.505199 13954 tablet_replica.cc:333] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:08:13.513226 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 87df3a9146bb41e1ac7468e17d2d9498 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:13.514521 12800 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:13.516173 13954 raft_consensus.cc:2243] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:13.516788 13954 raft_consensus.cc:2272] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:13.520040 12544 catalog_manager.cc:5654] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:13.525523 13946 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:13.546435 13954 ts_tablet_manager.cc:1916] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260202 15:08:13.548519 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 112417us
I20260202 15:08:13.550145 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ac5476ea30a46548db9ede417d946e6 with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:08:13.606611 12726 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:13.607223 12726 raft_consensus.cc:606] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
W20260202 15:08:13.611392 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 134148us
I20260202 15:08:13.616582 13980 tablet_replica.cc:333] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:08:13.617555 13980 raft_consensus.cc:2243] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:13.618893 13980 raft_consensus.cc:2272] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:13.624059 12799 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:13.624541 12799 raft_consensus.cc:606] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Received request to transfer leadership to TS bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:13.631713 13970 ts_tablet_manager.cc:1929] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:13.639673 13970 log.cc:1199] T c4798df17a1e4d18a594bb1b37b98bae P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/c4798df17a1e4d18a594bb1b37b98bae
I20260202 15:08:13.651816 13980 ts_tablet_manager.cc:1916] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:13.655926 12796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:13.656471 12796 raft_consensus.cc:606] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:13.658984 12644 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:13.659428 12644 raft_consensus.cc:606] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:13.660884 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet c4798df17a1e4d18a594bb1b37b98bae (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:13.663959 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260202 15:08:13.665452 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:13.666072 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:13.738896 13954 ts_tablet_manager.cc:1929] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:13.739434 13954 log.cc:1199] T 9335cdc844e44d608b9cdd8369f9afb1 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/9335cdc844e44d608b9cdd8369f9afb1
I20260202 15:08:13.740731 13929 raft_consensus.cc:993] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:13.741194 13832 raft_consensus.cc:1081] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:13.742331 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 9335cdc844e44d608b9cdd8369f9afb1 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:13.742583 13361 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44934
I20260202 15:08:13.743229 13361 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:13.743503 13361 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:13.744129 13980 ts_tablet_manager.cc:1929] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:13.744537 13980 log.cc:1199] T 2ac5476ea30a46548db9ede417d946e6 P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/2ac5476ea30a46548db9ede417d946e6
I20260202 15:08:13.746019 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet 2ac5476ea30a46548db9ede417d946e6 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:13.747627 13361 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:13.748952 13361 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:13.749902 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:13.750552 12644 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:13.750547 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:13.751277 12796 raft_consensus.cc:3055] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:13.751536 12796 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:13.752125 12796 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:13.753223 12796 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:13.756847 12644 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:13.757849 13326 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:13.758356 12796 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:13.758672 13977 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:13.759153 13977 raft_consensus.cc:697] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:13.759819 13977 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:13.776110 12547 catalog_manager.cc:5654] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:13.863723 13929 raft_consensus.cc:993] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:13.864289 13965 raft_consensus.cc:1081] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Signalling peer bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:13.865543 12644 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:34784
I20260202 15:08:13.865927 12644 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:13.866158 12644 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:13.871649 12644 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:13.872990 12644 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:13.874868 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:13.875319 12718 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:13.876139 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:13.876645 12795 raft_consensus.cc:3055] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:13.876847 12795 raft_consensus.cc:740] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:13.877329 12795 consensus_queue.cc:260] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:13.878078 12795 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:13.879992 12718 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:13.880738 12615 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:13.881383 13975 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:13.881775 13975 raft_consensus.cc:697] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:13.882285 12795 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:13.882472 13975 consensus_queue.cc:237] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:13.890529 12547 catalog_manager.cc:5654] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 3, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:14.139829 12795 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:14.140985 13971 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:14.141431 13975 raft_consensus.cc:993] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:14.141925 13975 raft_consensus.cc:1081] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:14.143365 12726 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:37626
I20260202 15:08:14.143795 12726 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:14.144120 12726 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:14.151091 12726 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:14.152887 12726 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:14.153959 12650 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:14.155937 13810 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:14.156472 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:14.156970 12795 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:14.163048 12795 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:14.163947 12690 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:14.164819 13976 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:14.165310 13976 raft_consensus.cc:697] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:14.166056 13976 consensus_queue.cc:237] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:14.171351 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:14.171859 12644 raft_consensus.cc:3055] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:14.172212 12644 raft_consensus.cc:740] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:14.172789 12644 consensus_queue.cc:260] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:14.175575 12644 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:14.185665 12644 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:14.187160 12545 catalog_manager.cc:5654] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:14.234021 13988 raft_consensus.cc:993] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:14.235639 13988 raft_consensus.cc:1081] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:14.236919 12796 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56734
I20260202 15:08:14.237313 12796 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:14.237635 12796 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:14.242108 12796 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:14.243995 12796 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:14.244292 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:14.244726 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:14.244800 12641 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:14.245230 12726 raft_consensus.cc:3055] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:14.245510 12726 raft_consensus.cc:740] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:14.246259 12726 consensus_queue.cc:260] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:14.247133 12726 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:14.250556 12641 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:14.251567 12726 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:14.252665 12763 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:14.253372 13965 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:14.253855 13965 raft_consensus.cc:697] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:14.257180 13965 consensus_queue.cc:237] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:14.266479 12544 catalog_manager.cc:5654] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:08:14.290261 12796 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:14.291323 13975 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:14.303881 12726 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:14.305846 13975 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:14.564424 12641 raft_consensus.cc:1275] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:14.565438 13988 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:14.576453 12796 raft_consensus.cc:1275] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:14.577908 13988 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260202 15:08:14.614738 12796 consensus_queue.cc:237] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:14.622066 12718 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:14.622264 12764 consensus_peers.cc:597] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebc29e67ee294975abcdfc9e53f8b6e1. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:14.623168 13832 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:14.623724 12644 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:14.624759 13832 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:14.629209 13965 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.630606 12718 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.632716 12644 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.645072 12544 catalog_manager.cc:5654] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.649434 12650 consensus_queue.cc:237] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:08:14.666363 12614 consensus_peers.cc:597] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30df58c1134a49e18c29202abd31da41. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:14.668548 12796 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:14.669116 12718 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:14.669582 13975 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:08:14.670674 13969 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:14.677345 13969 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.678746 12718 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.680985 12796 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.693382 12545 catalog_manager.cc:5654] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.698827 12650 consensus_queue.cc:237] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:14.706311 12796 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:14.706506 12718 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:14.706741 12614 consensus_peers.cc:597] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69e93e86b75b464fa4c55fdeb141acea. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:14.707418 13975 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:14.708161 13993 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:14.722256 12795 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.723876 12718 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.721032 13969 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.731724 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:14.738001 12725 consensus_queue.cc:237] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:14.743369 12544 catalog_manager.cc:5654] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.746178 12650 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:14.747217 12689 consensus_peers.cc:597] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85ea3f0239bb4603a3a363108154ef77. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:14.747581 13976 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:14.751451 12795 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:14.752900 13976 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:14.761097 13976 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.762351 12642 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.794467 12547 catalog_manager.cc:5654] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:14.799273 12796 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:14.813462 12725 raft_consensus.cc:1275] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:14.814761 13965 consensus_queue.cc:1048] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:14.820351 12644 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:14.821233 12644 raft_consensus.cc:606] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:14.826303 12718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:14.826766 12718 raft_consensus.cc:606] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Received request to transfer leadership to TS bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:14.828797 12799 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:14.829227 12799 raft_consensus.cc:606] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:14.830605 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260202 15:08:14.830940 12650 raft_consensus.cc:1275] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:14.832012 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:14.832615 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:14.837695 13965 consensus_queue.cc:1048] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:15.003721 13929 raft_consensus.cc:993] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:15.004240 13965 raft_consensus.cc:1081] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Signalling peer 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:15.005376 12718 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:37612
I20260202 15:08:15.005841 12718 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:15.006089 12718 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:15.010875 12718 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:15.012451 12718 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:15.013083 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:15.013624 13361 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:15.018426 13361 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:15.019084 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:15.019230 12689 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:15.019603 12796 raft_consensus.cc:3055] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:15.019897 12796 raft_consensus.cc:740] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:15.019842 13976 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:15.020553 13976 raft_consensus.cc:697] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:15.020627 12796 consensus_queue.cc:260] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:15.021374 12796 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:15.021222 13976 consensus_queue.cc:237] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:15.026330 12796 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:15.038339 12547 catalog_manager.cc:5654] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 2, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:15.076865 13969 raft_consensus.cc:993] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:15.077244 13975 raft_consensus.cc:1081] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:15.078363 12796 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56722
I20260202 15:08:15.078789 12796 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:15.079095 12796 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:15.088162 12796 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:15.088442 14014 ts_tablet_manager.cc:933] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:15.091375 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:15.091802 12650 raft_consensus.cc:3055] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:15.092018 12650 raft_consensus.cc:740] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:15.092088 12796 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:15.091547 14014 tablet_copy_client.cc:323] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:15.092604 12650 consensus_queue.cc:260] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:15.093394 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:15.093858 12718 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:15.093803 12650 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:15.095170 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 30df58c1134a49e18c29202abd31da41 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:15.095613 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 30df58c1134a49e18c29202abd31da41 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-30df58c1134a49e18c29202abd31da41
I20260202 15:08:15.100207 12650 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:15.101913 12660 tablet_copy_source_session.cc:215] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:15.104393 12763 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:15.106029 12718 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:15.107612 13965 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:15.108062 13965 raft_consensus.cc:697] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:15.108799 13965 consensus_queue.cc:237] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:15.113032 14014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30df58c1134a49e18c29202abd31da41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:15.120575 12547 catalog_manager.cc:5654] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:08:15.127522 14014 tablet_copy_client.cc:806] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:15.128002 14014 tablet_copy_client.cc:670] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:15.129184 14017 ts_tablet_manager.cc:933] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:15.131574 14014 tablet_copy_client.cc:538] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:15.138374 14014 tablet_bootstrap.cc:492] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:15.139724 14017 tablet_copy_client.cc:323] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:15.141108 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 69e93e86b75b464fa4c55fdeb141acea from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:15.141613 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 69e93e86b75b464fa4c55fdeb141acea from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-69e93e86b75b464fa4c55fdeb141acea
I20260202 15:08:15.149312 12660 tablet_copy_source_session.cc:215] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:15.154500 14019 ts_tablet_manager.cc:933] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:15.157215 14017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e93e86b75b464fa4c55fdeb141acea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:15.161442 14014 tablet_bootstrap.cc:492] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:15.162096 14014 tablet_bootstrap.cc:492] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:15.162565 14014 ts_tablet_manager.cc:1403] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.008s
I20260202 15:08:15.164832 14014 raft_consensus.cc:359] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:15.165513 14014 raft_consensus.cc:740] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:15.166061 14014 consensus_queue.cc:260] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:15.166914 14019 tablet_copy_client.cc:323] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.195:45829
I20260202 15:08:15.168388 12810 tablet_copy_service.cc:140] P 1eca64246cad41379cde4e2f9c3e25d2: Received BeginTabletCopySession request for tablet ebc29e67ee294975abcdfc9e53f8b6e1 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:32824)
I20260202 15:08:15.168900 12810 tablet_copy_service.cc:161] P 1eca64246cad41379cde4e2f9c3e25d2: Beginning new tablet copy session on tablet ebc29e67ee294975abcdfc9e53f8b6e1 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:32824: session id = 66b8c715b9e6479e9218b08bd3156380-ebc29e67ee294975abcdfc9e53f8b6e1
I20260202 15:08:15.169288 14014 ts_tablet_manager.cc:1434] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:08:15.171120 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-30df58c1134a49e18c29202abd31da41 received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:15.171470 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-30df58c1134a49e18c29202abd31da41 on tablet 30df58c1134a49e18c29202abd31da41 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:15.178324 12810 tablet_copy_source_session.cc:215] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:15.179665 14017 tablet_copy_client.cc:806] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:15.182340 14019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc29e67ee294975abcdfc9e53f8b6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:15.182353 14017 tablet_copy_client.cc:670] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:15.187243 14017 tablet_copy_client.cc:538] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:15.195374 14017 tablet_bootstrap.cc:492] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:15.196988 14019 tablet_copy_client.cc:806] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:15.197582 14019 tablet_copy_client.cc:670] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:15.200528 14019 tablet_copy_client.cc:538] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:15.206952 14019 tablet_bootstrap.cc:492] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:15.214073 14017 tablet_bootstrap.cc:492] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:15.214816 14017 tablet_bootstrap.cc:492] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:15.217154 14017 ts_tablet_manager.cc:1403] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.010s
I20260202 15:08:15.222133 13880 raft_consensus.cc:993] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:15.222561 13880 raft_consensus.cc:1081] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Signalling peer bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:15.223330 14017 raft_consensus.cc:359] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:15.224036 12644 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:34792
I20260202 15:08:15.224236 14017 raft_consensus.cc:740] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:15.224537 12644 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:15.224829 12644 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:15.224822 14017 consensus_queue.cc:260] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:15.229948 14017 ts_tablet_manager.cc:1434] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20260202 15:08:15.230414 12644 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:15.232690 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:15.233142 12725 raft_consensus.cc:3055] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:15.233425 12725 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:15.234233 14019 tablet_bootstrap.cc:492] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:15.234073 12725 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:15.235036 14019 tablet_bootstrap.cc:492] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:15.235141 12725 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:15.235594 14019 ts_tablet_manager.cc:1403] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.006s
I20260202 15:08:15.237958 14019 raft_consensus.cc:359] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:15.238574 14019 raft_consensus.cc:740] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:15.238812 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-69e93e86b75b464fa4c55fdeb141acea received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:15.239171 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-69e93e86b75b464fa4c55fdeb141acea on tablet 69e93e86b75b464fa4c55fdeb141acea with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:15.239109 14019 consensus_queue.cc:260] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:15.241524 14019 ts_tablet_manager.cc:1434] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260202 15:08:15.243453 12810 tablet_copy_service.cc:342] P 1eca64246cad41379cde4e2f9c3e25d2: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-ebc29e67ee294975abcdfc9e53f8b6e1 received from {username='slave'} at 127.0.0.1:32824
I20260202 15:08:15.243846 12810 tablet_copy_service.cc:434] P 1eca64246cad41379cde4e2f9c3e25d2: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-ebc29e67ee294975abcdfc9e53f8b6e1 on tablet ebc29e67ee294975abcdfc9e53f8b6e1 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:15.247517 12725 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:15.248060 12644 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:15.248378 12615 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:15.248835 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:15.249239 13975 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:15.249292 13360 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:15.249739 13975 raft_consensus.cc:697] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:15.250450 13975 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:15.255206 13360 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:15.260699 12544 catalog_manager.cc:5654] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 2 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:15.346292 14019 ts_tablet_manager.cc:933] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:15.355252 14019 tablet_copy_client.cc:323] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:08:15.356321 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 85ea3f0239bb4603a3a363108154ef77 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:08:15.356782 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 85ea3f0239bb4603a3a363108154ef77 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-85ea3f0239bb4603a3a363108154ef77
I20260202 15:08:15.365981 12736 tablet_copy_source_session.cc:215] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:15.372128 14019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ea3f0239bb4603a3a363108154ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:15.387190 14019 tablet_copy_client.cc:806] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:15.387653 14019 tablet_copy_client.cc:670] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:15.390388 14019 tablet_copy_client.cc:538] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:15.396303 14019 tablet_bootstrap.cc:492] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:15.417163 14019 tablet_bootstrap.cc:492] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:15.417860 14019 tablet_bootstrap.cc:492] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:15.418383 14019 ts_tablet_manager.cc:1403] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.006s
I20260202 15:08:15.420476 14019 raft_consensus.cc:359] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:15.420985 14019 raft_consensus.cc:740] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:15.421485 14019 consensus_queue.cc:260] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:15.423908 14019 ts_tablet_manager.cc:1434] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.005s user 0.005s sys 0.003s
I20260202 15:08:15.427084 12795 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:15.428409 13976 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:15.430593 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-85ea3f0239bb4603a3a363108154ef77 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:08:15.431003 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-85ea3f0239bb4603a3a363108154ef77 on tablet 85ea3f0239bb4603a3a363108154ef77 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:15.443925 13360 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:15.445077 13988 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:15.525584 12650 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:15.526788 13929 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:15.539268 12725 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:15.540716 13917 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:15.586658 13360 raft_consensus.cc:1217] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:15.610720 13360 raft_consensus.cc:1217] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:15.680019 13360 raft_consensus.cc:1217] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:15.860918 13360 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:15.863830 13969 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:15.866892 12726 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:15.868448 13993 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:15.875471 13361 raft_consensus.cc:1217] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:15.881167 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:15.939112 12795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:15.939607 12795 raft_consensus.cc:606] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Received request to transfer leadership to TS bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:15.941592 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:15.942111 12650 raft_consensus.cc:606] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:15.944792 12726 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:15.945243 12726 raft_consensus.cc:606] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:15.946506 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260202 15:08:15.947829 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:15.948560 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:16.007918 13965 raft_consensus.cc:1064] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:16.009017 13975 raft_consensus.cc:1064] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:16.009989 13965 consensus_queue.cc:237] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.010782 13975 consensus_queue.cc:237] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.015767 13360 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.016932 13361 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.017050 12726 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.016832 13993 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.018127 13929 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.019042 12650 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.019793 12725 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.021270 12795 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.022576 13965 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:08:16.023881 13929 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.022958 13975 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.025094 14040 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.034933 13929 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.038079 12644 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.045758 12726 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.046886 13360 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.064323 12545 catalog_manager.cc:5654] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.078786 13993 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.098888 13880 raft_consensus.cc:993] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:16.098515 13360 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.094272 12725 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.093055 12795 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.099291 13880 raft_consensus.cc:1081] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:16.118610 12795 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56734
I20260202 15:08:16.119069 12795 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:16.119377 12795 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:16.124076 12795 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:16.125476 12795 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:16.127439 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:16.127879 12718 raft_consensus.cc:3055] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:16.128139 12718 raft_consensus.cc:740] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:16.128705 12718 consensus_queue.cc:260] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:16.129410 12718 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:16.133652 12718 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:16.134611 12765 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:16.135370 14047 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:16.135753 14047 raft_consensus.cc:697] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:16.136448 14047 consensus_queue.cc:237] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:16.140367 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:16.140818 12650 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:16.138609 12547 catalog_manager.cc:5654] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.145888 12795 consensus_queue.cc:237] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.152892 13360 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:16.155179 13965 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:16.161273 12650 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:16.163977 13965 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.164896 13360 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.164013 12642 consensus_queue.cc:237] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.167656 12545 catalog_manager.cc:5654] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 2 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:08:16.168978 12718 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:16.172699 13361 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:16.171339 14047 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:16.175415 13975 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:16.180192 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet ebc29e67ee294975abcdfc9e53f8b6e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:16.181057 12546 catalog_manager.cc:5654] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.182155 12725 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:16.183003 12718 raft_consensus.cc:2955] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.185431 13993 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:16.190837 14040 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.195441 12725 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.198773 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc29e67ee294975abcdfc9e53f8b6e1 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:08:16.200670 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 30df58c1134a49e18c29202abd31da41 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:16.200670 13360 raft_consensus.cc:2955] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.208287 14053 tablet_replica.cc:333] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:08:16.209188 14053 raft_consensus.cc:2243] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:16.209847 14053 raft_consensus.cc:2272] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:16.209939 12547 catalog_manager.cc:5654] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 3 to 4, VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:16.226033 12789 tablet_service.cc:1552] Processing DeleteTablet for tablet 30df58c1134a49e18c29202abd31da41 with delete_type TABLET_DATA_TOMBSTONED (TS 1eca64246cad41379cde4e2f9c3e25d2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56714
I20260202 15:08:16.227408 14054 tablet_replica.cc:333] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2: stopping tablet replica
I20260202 15:08:16.229954 14053 ts_tablet_manager.cc:1916] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:16.232206 13993 raft_consensus.cc:1064] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:16.233235 14054 raft_consensus.cc:2243] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:16.234212 13993 consensus_queue.cc:237] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.234001 14054 raft_consensus.cc:2272] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:16.240999 14054 ts_tablet_manager.cc:1916] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:16.247025 14053 ts_tablet_manager.cc:1929] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:16.247459 14053 log.cc:1199] T ebc29e67ee294975abcdfc9e53f8b6e1 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/ebc29e67ee294975abcdfc9e53f8b6e1
I20260202 15:08:16.249012 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet ebc29e67ee294975abcdfc9e53f8b6e1 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:16.249861 13360 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.251230 13975 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.254907 14054 ts_tablet_manager.cc:1929] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:16.255309 14054 log.cc:1199] T 30df58c1134a49e18c29202abd31da41 P 1eca64246cad41379cde4e2f9c3e25d2: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-2-root/wals/30df58c1134a49e18c29202abd31da41
I20260202 15:08:16.260933 12533 catalog_manager.cc:4985] TS 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): tablet 30df58c1134a49e18c29202abd31da41 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:16.262436 12795 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.265247 14040 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.258658 13969 raft_consensus.cc:993] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:16.261011 12725 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.268491 14040 raft_consensus.cc:1081] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:16.270364 12796 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56722
I20260202 15:08:16.270850 12796 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:16.271188 12796 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:16.270479 14040 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:08:16.279425 12796 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.284123 12796 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:16.284955 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:16.285769 13361 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:16.289633 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:16.290119 12650 raft_consensus.cc:3055] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:16.290316 12650 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:16.290827 12650 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.291558 12650 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:16.296626 12650 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:16.297842 13361 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:16.298323 12763 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:16.299450 13965 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:16.299865 13965 raft_consensus.cc:697] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:16.300593 13965 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.302510 14040 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.304044 12795 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.310943 12725 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.315100 13361 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.321071 12545 catalog_manager.cc:5654] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:16.326795 12546 catalog_manager.cc:5654] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:16.336812 12644 consensus_queue.cc:237] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.341176 13361 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:16.341835 12796 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:16.345430 14040 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:16.347885 13993 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:16.352881 13880 raft_consensus.cc:1064] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:16.355945 13880 consensus_queue.cc:237] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.355060 14058 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.355937 12796 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.360395 13361 raft_consensus.cc:2955] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.378123 12531 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 69e93e86b75b464fa4c55fdeb141acea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:16.378036 12546 catalog_manager.cc:5654] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) evicted. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.389392 12644 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.389940 12795 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.392985 13361 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:16.395970 13988 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.396631 13976 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.397272 12710 tablet_service.cc:1552] Processing DeleteTablet for tablet 69e93e86b75b464fa4c55fdeb141acea with delete_type TABLET_DATA_TOMBSTONED (TS 6921687ed3814cccb2bd9c606875b9f0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37604
I20260202 15:08:16.397648 13880 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:08:16.415457 14047 raft_consensus.cc:993] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:16.415910 14047 raft_consensus.cc:1081] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Signalling peer bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:16.420727 12650 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:34784
I20260202 15:08:16.421279 12650 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:16.421648 12650 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:16.423274 13988 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.424297 12795 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.427281 12650 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:16.429252 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:16.429702 12725 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:16.429878 12650 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:16.434501 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:16.435156 12796 raft_consensus.cc:3055] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:16.436257 12725 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:16.436533 12796 raft_consensus.cc:740] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:16.437232 12796 consensus_queue.cc:260] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:16.438014 12796 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:16.439541 12615 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:16.440372 14058 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:16.440764 14058 raft_consensus.cc:697] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:16.441409 14058 consensus_queue.cc:237] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:16.441725 12547 catalog_manager.cc:5654] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.467305 12545 catalog_manager.cc:5654] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 3, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:16.474022 14063 tablet_replica.cc:333] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0: stopping tablet replica
I20260202 15:08:16.475589 14063 raft_consensus.cc:2243] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:16.476315 14063 raft_consensus.cc:2272] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:16.479377 13361 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.480563 14063 ts_tablet_manager.cc:1916] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:16.478596 12650 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.487723 12796 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:16.505679 14063 ts_tablet_manager.cc:1929] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:16.506074 14063 log.cc:1199] T 69e93e86b75b464fa4c55fdeb141acea P 6921687ed3814cccb2bd9c606875b9f0: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-1-root/wals/69e93e86b75b464fa4c55fdeb141acea
I20260202 15:08:16.508689 12722 consensus_queue.cc:237] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:16.507771 12533 catalog_manager.cc:4985] TS 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): tablet 69e93e86b75b464fa4c55fdeb141acea (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:16.512651 13361 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:16.512781 12796 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260202 15:08:16.513765 13976 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:16.514798 13988 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:16.522235 12796 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.522408 13361 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.534027 12546 catalog_manager.cc:5654] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New cstate: current_term: 1 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.520753 13988 raft_consensus.cc:2955] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:16.546612 12533 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 85ea3f0239bb4603a3a363108154ef77 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260202 15:08:16.548987 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet 85ea3f0239bb4603a3a363108154ef77 with delete_type TABLET_DATA_TOMBSTONED (TS bbf140958c3b4f339c512a9dc4be2dab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34778
I20260202 15:08:16.549700 14053 tablet_replica.cc:333] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab: stopping tablet replica
I20260202 15:08:16.550493 14053 raft_consensus.cc:2243] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:08:16.551098 14053 raft_consensus.cc:2272] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:08:16.553687 14053 ts_tablet_manager.cc:1916] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260202 15:08:16.562404 14053 ts_tablet_manager.cc:1929] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260202 15:08:16.562638 14053 log.cc:1199] T 85ea3f0239bb4603a3a363108154ef77 P bbf140958c3b4f339c512a9dc4be2dab: Deleting WAL directory at /tmp/dist-test-taskehV2zm/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1770044797646468-11451-0/minicluster-data/ts-0-root/wals/85ea3f0239bb4603a3a363108154ef77
I20260202 15:08:16.563671 12531 catalog_manager.cc:4985] TS bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): tablet 85ea3f0239bb4603a3a363108154ef77 (table test-workload [id=77056d7cc1f7453cb5d987f5948bb9ef]) successfully deleted
I20260202 15:08:16.625321 12650 raft_consensus.cc:1275] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:16.626278 13929 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.635861 12725 raft_consensus.cc:1275] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:16.637305 13965 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.820675 12650 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:16.821866 13965 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:16.828907 13361 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:16.829905 13965 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:16.905977 12796 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:16.907152 12722 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:16.907312 13993 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.908515 13993 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:16.988802 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:17.018381 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:17.018803 12650 raft_consensus.cc:606] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:17.020766 12796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:17.021081 12796 raft_consensus.cc:606] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Received request to transfer leadership to TS bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:17.022986 12796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f72d171c95a149d890432428ecf80739"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:17.023445 12796 raft_consensus.cc:606] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:17.024462 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260202 15:08:17.025290 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:17.025710 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:17.137462 13965 raft_consensus.cc:993] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:17.137929 13929 raft_consensus.cc:1081] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Signalling peer bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:17.139082 12650 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:34784
I20260202 15:08:17.139446 12650 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:17.139679 12650 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:17.145358 12650 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.147207 12650 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:17.148152 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:17.148540 12725 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:17.149653 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:17.150134 12795 raft_consensus.cc:3055] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:17.150308 12795 raft_consensus.cc:740] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:17.150897 12795 consensus_queue.cc:260] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.151629 12795 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:17.152428 12725 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:17.153290 12615 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:17.153930 14040 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:17.154364 14040 raft_consensus.cc:697] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:17.155081 14040 consensus_queue.cc:237] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.156833 12795 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:17.166172 12546 catalog_manager.cc:5654] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 3, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:17.212723 13965 raft_consensus.cc:993] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:17.213073 13929 raft_consensus.cc:1081] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Signalling peer 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:17.214172 12725 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f72d171c95a149d890432428ecf80739"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:37612
I20260202 15:08:17.214577 12725 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:17.214819 12725 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:17.218348 12725 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.219444 12725 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:17.220124 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:17.220278 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:17.220634 12650 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:17.220757 12795 raft_consensus.cc:3055] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:17.221110 12795 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:17.221619 12795 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.222416 12795 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:17.224805 12650 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:17.225582 12688 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:17.226099 13988 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:17.226438 12795 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:17.227156 13988 raft_consensus.cc:697] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:17.228000 13988 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.235224 12546 catalog_manager.cc:5654] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 2 to 3, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 3 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:17.346407 14058 raft_consensus.cc:993] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:17.346822 13993 raft_consensus.cc:1081] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Signalling peer 1eca64246cad41379cde4e2f9c3e25d2 to start an election
I20260202 15:08:17.347831 12795 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:56722
I20260202 15:08:17.348196 12795 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:17.348393 12795 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:17.352061 12795 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.353492 12795 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:17.353940 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:17.354580 12642 raft_consensus.cc:3055] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:17.354862 12642 raft_consensus.cc:740] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:17.353982 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:17.355495 12642 consensus_queue.cc:260] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.356092 12722 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:17.356567 12642 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:17.359791 12722 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:17.360563 12765 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:17.361214 13965 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:17.361568 13965 raft_consensus.cc:697] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:17.361756 12642 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:17.362226 13965 consensus_queue.cc:237] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.369076 12546 catalog_manager.cc:5654] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 3 to 4, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 4 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:08:17.543069 12725 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:17.544312 13993 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:17.558576 12795 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:17.564546 14040 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260202 15:08:17.622979 12596 auto_rebalancer.cc:256] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20260202 15:08:17.626125 12596 auto_rebalancer.cc:726] Could not move replica: Incomplete: tablet f72d171c95a149d890432428ecf80739, TS 1eca64246cad41379cde4e2f9c3e25d2 -> TS 66b8c715b9e6479e9218b08bd3156380 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:17.626344 12596 auto_rebalancer.cc:266] scheduled replica move failed to complete: Incomplete: tablet f72d171c95a149d890432428ecf80739, TS 1eca64246cad41379cde4e2f9c3e25d2 -> TS 66b8c715b9e6479e9218b08bd3156380 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:17.629207 12596 auto_rebalancer.cc:726] Could not move replica: Incomplete: tablet 83fc6cc808d9450788aeecc86caa94bd, TS 6921687ed3814cccb2bd9c606875b9f0 -> TS 66b8c715b9e6479e9218b08bd3156380 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:17.629410 12596 auto_rebalancer.cc:266] scheduled replica move failed to complete: Incomplete: tablet 83fc6cc808d9450788aeecc86caa94bd, TS 6921687ed3814cccb2bd9c606875b9f0 -> TS 66b8c715b9e6479e9218b08bd3156380 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:17.631925 12596 auto_rebalancer.cc:726] Could not move replica: Incomplete: tablet 62adc5e06e9d428f94bdbd03c58c5e5b, TS bbf140958c3b4f339c512a9dc4be2dab -> TS 66b8c715b9e6479e9218b08bd3156380 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:17.632128 12596 auto_rebalancer.cc:266] scheduled replica move failed to complete: Incomplete: tablet 62adc5e06e9d428f94bdbd03c58c5e5b, TS bbf140958c3b4f339c512a9dc4be2dab -> TS 66b8c715b9e6479e9218b08bd3156380 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:17.635454 12596 auto_rebalancer.cc:726] Could not move replica: Incomplete: tablet 21bc61b2f0f64d0b80e5d98615b1d6b6, TS 1eca64246cad41379cde4e2f9c3e25d2 -> TS 66b8c715b9e6479e9218b08bd3156380 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:17.635617 12596 auto_rebalancer.cc:266] scheduled replica move failed to complete: Incomplete: tablet 21bc61b2f0f64d0b80e5d98615b1d6b6, TS 1eca64246cad41379cde4e2f9c3e25d2 -> TS 66b8c715b9e6479e9218b08bd3156380 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:17.680399 12796 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:17.681488 14082 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:17.694615 12650 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:17.696036 13988 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:08:17.770465 12642 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:08:17.771840 13965 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:17.778033 12722 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:08:17.779285 14092 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:18.060989 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:18.096686 12795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1c6861781ee34db9b79870494af1cced"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:18.097317 12795 raft_consensus.cc:606] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:18.099567 12722 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:18.100045 12722 raft_consensus.cc:606] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:18.102209 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:18.102593 12650 raft_consensus.cc:606] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:18.103615 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260202 15:08:18.104702 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:18.105228 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:18.273589 14082 raft_consensus.cc:993] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:18.274029 13988 raft_consensus.cc:1081] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:18.275213 13361 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44940
I20260202 15:08:18.275573 13361 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:18.275818 13361 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:18.281392 13361 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:18.282642 13361 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:18.283165 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:18.283486 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:18.283773 12722 raft_consensus.cc:3055] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:18.284137 12722 raft_consensus.cc:740] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:18.284144 12795 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:18.284862 12722 consensus_queue.cc:260] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:18.285704 12722 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:18.288157 12795 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:18.288908 13328 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:18.289422 14095 raft_consensus.cc:2804] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:18.289567 12722 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:18.289862 14095 raft_consensus.cc:697] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:18.290530 14095 consensus_queue.cc:237] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:18.297621 12546 catalog_manager.cc:5654] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:18.391892 13993 raft_consensus.cc:993] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:18.392792 13993 raft_consensus.cc:1081] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:18.398897 12725 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:37626
I20260202 15:08:18.404819 12725 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:18.407344 12725 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:18.417375 12725 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:18.419503 12725 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:18.420553 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:18.421011 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:18.422150 12642 raft_consensus.cc:3055] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:18.422411 13361 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:18.422495 12642 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:18.423985 12642 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:18.425815 12642 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:18.430539 13361 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:18.431758 12689 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:18.432586 13988 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:18.433005 13988 raft_consensus.cc:697] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:18.433794 12642 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:18.434093 13988 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:18.458104 12546 catalog_manager.cc:5654] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:18.481202 13965 raft_consensus.cc:993] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:18.482038 13965 raft_consensus.cc:1081] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Signalling peer bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:18.489804 12642 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1c6861781ee34db9b79870494af1cced"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:34784
I20260202 15:08:18.490409 12642 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:18.490825 12642 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:18.498720 12642 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:18.501067 12642 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:18.501837 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:18.502521 12722 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:18.504010 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:18.504714 12795 raft_consensus.cc:3055] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:18.505059 12795 raft_consensus.cc:740] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:18.506016 12795 consensus_queue.cc:260] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:18.507704 12795 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:18.514149 12722 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:18.516320 12795 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:18.517005 12615 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:18.518568 14040 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:18.519169 14040 raft_consensus.cc:697] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:18.520285 14040 consensus_queue.cc:237] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:18.538488 12546 catalog_manager.cc:5654] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 2, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 2 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:18.692812 12642 consensus_queue.cc:237] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:18.697685 12796 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260202 15:08:18.698081 12614 consensus_peers.cc:597] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78ef01280730461580740297403fe090. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:18.698194 12722 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:08:18.698853 14094 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:18.699591 14040 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:18.707056 14040 raft_consensus.cc:2955] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:18.708916 12722 raft_consensus.cc:2955] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:18.712685 12796 raft_consensus.cc:2955] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:18.720826 12642 consensus_queue.cc:237] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:18.722770 12546 catalog_manager.cc:5654] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 2, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:18.742982 12796 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260202 15:08:18.742997 12614 consensus_peers.cc:597] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fae829b1c574bc795c5a424036329b3. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:18.745278 12725 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260202 15:08:18.746150 14094 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:18.747012 14058 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:18.753311 14058 raft_consensus.cc:2955] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:18.754921 12722 raft_consensus.cc:2955] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:18.758323 12796 raft_consensus.cc:2955] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:18.767599 12545 catalog_manager.cc:5654] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 4, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:18.767932 12722 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:18.774941 12642 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260202 15:08:18.775342 12796 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260202 15:08:18.775789 12689 consensus_peers.cc:597] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f72d171c95a149d890432428ecf80739. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:18.776136 13988 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:18.783102 13988 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260202 15:08:18.865711 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 73307.8us
I20260202 15:08:18.880645 13361 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:18.882002 12795 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:18.886837 14095 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:18.897977 14082 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:18.904801 12722 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:18.943707 14009 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:18.945207 14082 raft_consensus.cc:2955] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:18.946388 12642 raft_consensus.cc:2955] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:18.952564 12650 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:18.955127 14102 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260202 15:08:18.983784 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 45092.6us
I20260202 15:08:18.989254 12799 raft_consensus.cc:1275] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:18.990582 14115 consensus_queue.cc:1048] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:18.991423 12726 consensus_queue.cc:237] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:19.043068 12547 catalog_manager.cc:5654] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index -1 to 4, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 3 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:19.047096 12796 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260202 15:08:19.088361 12796 raft_consensus.cc:2955] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
W20260202 15:08:19.095042 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 46659.3us
I20260202 15:08:19.095842 14130 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:19.096426 12726 raft_consensus.cc:1275] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260202 15:08:19.099066 12689 consensus_peers.cc:597] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 -> Peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Couldn't send request to peer 66b8c715b9e6479e9218b08bd3156380. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 015f643434d84eac8a3774b663eb4220. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:08:19.101591 14115 consensus_queue.cc:1048] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:19.103032 12644 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260202 15:08:19.104297 14090 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:19.128775 14102 raft_consensus.cc:2955] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:19.137198 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:19.138355 12650 raft_consensus.cc:2955] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:19.160552 12545 catalog_manager.cc:5654] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: config changed from index -1 to 3, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:19.173647 12796 raft_consensus.cc:2955] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:19.180234 14136 ts_tablet_manager.cc:933] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:19.192014 12726 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:19.192198 14136 tablet_copy_client.cc:323] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:19.192548 12726 raft_consensus.cc:606] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:19.194293 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 9fae829b1c574bc795c5a424036329b3 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:19.194875 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 9fae829b1c574bc795c5a424036329b3 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-9fae829b1c574bc795c5a424036329b3
I20260202 15:08:19.198732 12642 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:19.199214 12642 raft_consensus.cc:606] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:19.200342 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260202 15:08:19.201686 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:19.202502 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:19.206527 12660 tablet_copy_source_session.cc:215] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:19.210682 14136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fae829b1c574bc795c5a424036329b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:19.223894 14138 ts_tablet_manager.cc:933] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:19.228674 14136 tablet_copy_client.cc:806] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:19.229183 14136 tablet_copy_client.cc:670] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:19.234532 14138 tablet_copy_client.cc:323] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:08:19.236142 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet f72d171c95a149d890432428ecf80739 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:08:19.236634 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet f72d171c95a149d890432428ecf80739 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-f72d171c95a149d890432428ecf80739
I20260202 15:08:19.239914 14136 tablet_copy_client.cc:538] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:19.245152 12736 tablet_copy_source_session.cc:215] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:19.248361 14138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f72d171c95a149d890432428ecf80739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:19.251724 14136 tablet_bootstrap.cc:492] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:19.264292 14138 tablet_copy_client.cc:806] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:19.264801 14138 tablet_copy_client.cc:670] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:19.268316 14138 tablet_copy_client.cc:538] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:19.274528 14136 tablet_bootstrap.cc:492] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:19.275426 14136 tablet_bootstrap.cc:492] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:19.276085 14136 ts_tablet_manager.cc:1403] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.003s
I20260202 15:08:19.276139 14138 tablet_bootstrap.cc:492] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:19.279806 14136 raft_consensus.cc:359] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:19.280887 14136 raft_consensus.cc:740] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:19.281582 14136 consensus_queue.cc:260] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:19.291280 14136 ts_tablet_manager.cc:1434] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260202 15:08:19.292254 14141 ts_tablet_manager.cc:933] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:19.292666 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-9fae829b1c574bc795c5a424036329b3 received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:19.293169 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-9fae829b1c574bc795c5a424036329b3 on tablet 9fae829b1c574bc795c5a424036329b3 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:19.304597 14141 tablet_copy_client.cc:323] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.193:40057
I20260202 15:08:19.305872 12660 tablet_copy_service.cc:140] P bbf140958c3b4f339c512a9dc4be2dab: Received BeginTabletCopySession request for tablet 78ef01280730461580740297403fe090 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:42782)
I20260202 15:08:19.306232 12660 tablet_copy_service.cc:161] P bbf140958c3b4f339c512a9dc4be2dab: Beginning new tablet copy session on tablet 78ef01280730461580740297403fe090 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:42782: session id = 66b8c715b9e6479e9218b08bd3156380-78ef01280730461580740297403fe090
I20260202 15:08:19.308835 14138 tablet_bootstrap.cc:492] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:19.310720 14138 tablet_bootstrap.cc:492] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:19.311744 14138 ts_tablet_manager.cc:1403] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.036s user 0.025s sys 0.002s
I20260202 15:08:19.312186 12660 tablet_copy_source_session.cc:215] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:19.316713 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ef01280730461580740297403fe090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:19.316431 14138 raft_consensus.cc:359] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:19.317178 14138 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:19.317842 14138 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:19.323182 14138 ts_tablet_manager.cc:1434] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260202 15:08:19.324643 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-f72d171c95a149d890432428ecf80739 received from {username='slave'} at 127.0.0.1:59480
I20260202 15:08:19.324990 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-f72d171c95a149d890432428ecf80739 on tablet f72d171c95a149d890432428ecf80739 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:19.331754 14102 raft_consensus.cc:993] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:19.332126 14090 raft_consensus.cc:1081] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:19.332639 14141 tablet_copy_client.cc:806] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:19.333119 14141 tablet_copy_client.cc:670] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:19.333550 13361 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44940
I20260202 15:08:19.334038 13361 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:19.334360 13361 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:19.336699 14141 tablet_copy_client.cc:538] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:19.340233 13361 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:19.341835 13361 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:19.342353 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:19.342581 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:19.343146 12642 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:19.344449 14141 tablet_bootstrap.cc:492] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:19.343132 12722 raft_consensus.cc:3055] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:19.345127 12722 raft_consensus.cc:740] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:19.345808 12722 consensus_queue.cc:260] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:19.348217 12722 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:19.350157 12642 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:19.351054 13326 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:19.351696 14009 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:19.352056 14009 raft_consensus.cc:697] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:19.352679 14009 consensus_queue.cc:237] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:19.354038 12722 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:19.368364 14141 tablet_bootstrap.cc:492] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:19.369105 14141 tablet_bootstrap.cc:492] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:19.369614 14141 ts_tablet_manager.cc:1403] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.001s
I20260202 15:08:19.372946 14141 raft_consensus.cc:359] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:19.373740 14141 raft_consensus.cc:740] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:19.374498 14141 consensus_queue.cc:260] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:19.377101 14141 ts_tablet_manager.cc:1434] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260202 15:08:19.378934 12660 tablet_copy_service.cc:342] P bbf140958c3b4f339c512a9dc4be2dab: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-78ef01280730461580740297403fe090 received from {username='slave'} at 127.0.0.1:42782
I20260202 15:08:19.379273 12660 tablet_copy_service.cc:434] P bbf140958c3b4f339c512a9dc4be2dab: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-78ef01280730461580740297403fe090 on tablet 78ef01280730461580740297403fe090 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:19.382481 12546 catalog_manager.cc:5654] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:19.553045 14040 raft_consensus.cc:993] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:19.553543 14058 raft_consensus.cc:1081] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Signalling peer 6921687ed3814cccb2bd9c606875b9f0 to start an election
I20260202 15:08:19.558393 12722 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:37626
I20260202 15:08:19.558849 12722 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:19.559130 12722 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:19.566188 12722 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:19.567509 12722 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:19.568217 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:19.568226 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:19.568800 12796 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:19.568835 12642 raft_consensus.cc:3055] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:19.569191 12642 raft_consensus.cc:740] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:19.569756 12642 consensus_queue.cc:260] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:19.570765 12642 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:19.574345 12796 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:19.576363 14141 ts_tablet_manager.cc:933] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Initiating tablet copy from peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:19.576340 12690 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:19.577106 14130 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:19.577900 14141 tablet_copy_client.cc:323] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Beginning tablet copy session from remote peer at address 127.11.46.194:44367
I20260202 15:08:19.579236 12642 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:19.579504 14130 raft_consensus.cc:697] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:19.579942 12736 tablet_copy_service.cc:140] P 6921687ed3814cccb2bd9c606875b9f0: Received BeginTabletCopySession request for tablet 015f643434d84eac8a3774b663eb4220 from peer 66b8c715b9e6479e9218b08bd3156380 ({username='slave'} at 127.0.0.1:59480)
I20260202 15:08:19.580403 12736 tablet_copy_service.cc:161] P 6921687ed3814cccb2bd9c606875b9f0: Beginning new tablet copy session on tablet 015f643434d84eac8a3774b663eb4220 from peer 66b8c715b9e6479e9218b08bd3156380 at {username='slave'} at 127.0.0.1:59480: session id = 66b8c715b9e6479e9218b08bd3156380-015f643434d84eac8a3774b663eb4220
I20260202 15:08:19.580404 14130 consensus_queue.cc:237] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:19.586346 12736 tablet_copy_source_session.cc:215] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:08:19.591195 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015f643434d84eac8a3774b663eb4220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:08:19.592146 12546 catalog_manager.cc:5654] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:19.605370 14141 tablet_copy_client.cc:806] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 0 data blocks...
I20260202 15:08:19.606493 14141 tablet_copy_client.cc:670] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Starting download of 1 WAL segments...
I20260202 15:08:19.612438 14141 tablet_copy_client.cc:538] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:08:19.620316 14141 tablet_bootstrap.cc:492] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap starting.
I20260202 15:08:19.669207 14141 tablet_bootstrap.cc:492] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:08:19.669929 14141 tablet_bootstrap.cc:492] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Bootstrap complete.
I20260202 15:08:19.670629 14141 ts_tablet_manager.cc:1403] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Time spent bootstrapping tablet: real 0.050s user 0.018s sys 0.013s
I20260202 15:08:19.673115 14141 raft_consensus.cc:359] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:19.673889 14141 raft_consensus.cc:740] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Initialized, Role: LEARNER
I20260202 15:08:19.674463 14141 consensus_queue.cc:260] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:19.677175 14141 ts_tablet_manager.cc:1434] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:08:20.279458 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260202 15:08:20.708213 12747 debug-util.cc:398] Leaking SignalData structure 0x7b0800166380 after lost signal to thread 11454
W20260202 15:08:20.723526 12764 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.755s user 0.000s sys 0.018s
W20260202 15:08:20.724020 12764 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s user 0.000s sys 0.018s
I20260202 15:08:20.735965 14151 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.736529 14151 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.739238 14151 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.741765 14150 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:20.742199 14150 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.744138 14150 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.746479 14150 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:20.746923 14150 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.748795 14150 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:20.756548 14152 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.763008 14152 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.764528 14150 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:20.765085 14150 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.766932 14150 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:20.768297 14152 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.781798 14152 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:20.782783 14152 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.787191 14153 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.788415 14153 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.796039 14152 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:20.802863 14153 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.812135 12546 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39704 (request call id 234) took 1130 ms. Trace:
I20260202 15:08:20.814429 12546 rpcz_store.cc:276] 0202 15:08:19.681869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:20.757455 (+1075586us) service_pool.cc:225] Handling call
0202 15:08:20.812062 (+ 54607us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":16768}
I20260202 15:08:20.813541 14158 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:20.821024 14158 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.821301 14153 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:20.822858 14155 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:20.823616 14155 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.824888 14158 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.822767 14153 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.828519 14153 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.830201 14167 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.830415 14155 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.830719 14167 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.831091 14102 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.831773 14102 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.832751 14167 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.834159 14152 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:20.835004 14040 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:20.835587 14040 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.837105 14040 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.840808 14154 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.841224 14154 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.835089 14152 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:20.842854 14154 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.844146 14152 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:20.848894 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:20.850111 12650 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:20.859685 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:20.861006 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:20.864787 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:20.866354 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:20.869508 12763 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:20.870239 14168 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:20.870569 14151 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:20.870934 14151 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.870661 14168 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:20.877952 14168 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.876147 14151 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:20.887516 14158 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.887935 14158 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.889410 14158 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.889777 14158 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.890159 14158 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.894186 14130 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.894906 14130 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.891399 14102 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.896838 14163 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:20.897936 14130 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.902109 14130 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.903676 14130 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.905601 14009 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:20.906014 14009 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.906630 14151 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.907509 14009 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:20.909030 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:20.909504 12650 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:20.898620 14102 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.910337 14102 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.897408 14163 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.904956 14153 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.911864 14102 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.911687 14153 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.913079 14154 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.913564 14154 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.913821 14153 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.905313 14168 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.934779 14154 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.935143 14101 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.935585 14101 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.936627 14154 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:20.937565 14151 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.939465 12650 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:20.940770 14130 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.927304 14168 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.944236 14171 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:20.944381 12763 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:20.944867 14171 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.946095 14176 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.947386 14171 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:20.947237 14176 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.946738 14151 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:20.937790 14101 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:20.948680 14163 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.948977 14158 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.949059 14168 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.941507 14154 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.967415 14171 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:20.970233 14177 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.965436 14153 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.975834 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:20.978515 12650 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 1.
I20260202 15:08:20.967813 14151 raft_consensus.cc:697] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:20.968926 14172 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:20.980465 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:20.969208 14101 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.979926 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:20.980801 14113 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:20.968885 14171 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.984337 14113 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.986624 14152 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:20.981453 14102 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.985829 14168 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:20.981417 14153 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.024701 14113 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.027591 14058 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.028260 14058 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.030278 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.032200 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.030332 14058 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:20.995411 14150 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.039036 14040 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.041270 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.040323 14102 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.988963 14101 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:21.046927 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 60080.7us
I20260202 15:08:21.045229 14040 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.047164 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:20.995280 14152 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:20.987890 14172 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:20.981464 14177 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.055519 14171 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.057475 14102 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.056665 14040 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.059372 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:21.044837 14178 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.063444 14040 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.055151 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.067555 14186 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.068028 14186 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.069415 14040 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.047013 14058 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.046733 14151 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.056059 14153 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.075273 14040 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.054869 14154 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.054638 14180 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.074213 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.047060 14168 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.047857 14150 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.083499 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.076005 14058 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.079564 14183 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.091188 14179 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.095302 14185 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.097931 12641 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:21.095607 14090 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.081696 14153 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.085346 14101 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:21.098580 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 115602us
I20260202 15:08:21.076704 14178 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.096000 14176 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.102093 13326 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.103245 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.103734 12795 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:21.095304 14184 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.102526 14130 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.102988 14101 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.117671 14187 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.118009 13361 raft_consensus.cc:1217] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260202 15:08:21.118115 14187 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.120448 14192 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.121110 14187 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.124696 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.113112 14188 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.125429 12799 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:21.111353 14191 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.121079 14192 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.081020 14180 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.102007 14153 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.125499 14188 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.101702 14183 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.106299 14179 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.126230 14191 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.105000 14185 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.106679 14090 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.255478 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.255949 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.256480 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.256672 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.257028 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.257637 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.257864 14183 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.258312 14183 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.258514 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.152418 14154 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:21.258744 14183 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.259169 14154 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.259681 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.259080 12794 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:21.260344 13356 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:21.260603 14183 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.150929 14101 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:21.235338 12747 debug-util.cc:398] Leaking SignalData structure 0x7b0800167a20 after lost signal to thread 12582
I20260202 15:08:21.261286 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:21.262071 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08001eb940 after lost signal to thread 12674
I20260202 15:08:21.262158 14101 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:21.262550 12791 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:21.262864 14101 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.263177 12615 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.111665 14176 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.263401 14101 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.263850 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.122036 14130 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.265244 14101 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.265492 13355 raft_consensus.cc:1217] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20260202 15:08:21.265890 14130 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.266023 14191 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.266525 14191 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.163023 14190 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.182493 14186 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.267695 14090 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.152283 14152 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:21.266888 14191 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.268646 14191 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.268841 14188 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.262286 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.150393 14184 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.151940 14177 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.270200 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.269774 14180 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.270681 12796 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:21.170306 14150 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:21.196641 14164 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.184201 14172 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:21.167295 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.271816 14164 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.177632 14174 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.272691 14184 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.273088 14174 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.273787 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.274219 14164 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.218472 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.275070 14174 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.183676 14178 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.159689 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.276705 14179 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:21.270414 13354 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:21.167248 14195 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.185045 14168 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.277541 14195 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.214962 14171 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.264012 14176 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.233911 14173 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.279176 14176 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.279546 14195 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.279326 14173 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:21.116166 14102 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.214771 14158 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.280469 14102 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.280776 14158 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.282111 14102 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.151839 14187 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.182387 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.283039 14187 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.267374 14190 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.171348 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.288197 13357 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:21.284047 14158 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.291237 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.141659 14113 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.294093 14113 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:21.295277 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.295873 12800 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:21.297816 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.220345 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.278448 14171 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.285398 12722 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:21.299788 12722 raft_consensus.cc:606] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Received request to transfer leadership to TS bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.299944 12641 raft_consensus.cc:1275] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:21.301784 14171 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.302431 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.304078 14153 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.178125 14058 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:21.275566 12747 debug-util.cc:398] Leaking SignalData structure 0x7b0800329dc0 after lost signal to thread 12750
I20260202 15:08:21.129786 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.286051 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.128840 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.286497 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.287571 12615 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.177733 14166 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.237207 14193 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.288904 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.289140 14190 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:21.299336 14192 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.318114 12650 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:21.316941 12647 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:21.281374 14173 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.332281 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260202 15:08:21.338439 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.305590 12764 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.339748 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:21.340485 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:21.343004 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.291023 14185 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.343540 13361 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:21.346503 12764 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:21.347417 14178 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.347774 14178 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.348134 14178 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.351879 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.295874 14113 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:21.361377 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.361801 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.336845 14155 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260202 15:08:21.362129 12747 debug-util.cc:398] Leaking SignalData structure 0x7b080033e5a0 after lost signal to thread 12824
I20260202 15:08:21.352571 14190 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:21.362008 13357 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:21.362430 14155 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.355217 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.377084 12763 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:21.367831 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.362391 13359 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:21.369181 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.395074 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:21.395911 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 56312.2us
I20260202 15:08:21.357401 14191 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.397323 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.354012 14166 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.400629 12764 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:21.395906 13363 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:21.397782 12725 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:21.402145 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.408602 14205 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.384630 14102 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.383591 14155 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.411281 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.383062 14174 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.412953 12688 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.413148 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.417420 12688 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.413239 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.414294 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.418210 12722 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
W20260202 15:08:21.401594 12747 debug-util.cc:398] Leaking SignalData structure 0x7b080035bd40 after lost signal to thread 13387
I20260202 15:08:21.411841 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.396972 14179 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.416112 14102 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.419580 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.400132 14151 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.416324 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.420508 12615 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.419075 12645 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:21.424052 12615 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.416249 14201 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.412985 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.428112 14040 raft_consensus.cc:2804] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.429373 14040 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.429772 14040 raft_consensus.cc:3060] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.430512 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.428872 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.408776 14200 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:21.433409 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 95631.9us
I20260202 15:08:21.432736 12615 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.432663 12718 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:21.421938 14183 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:21.432494 12747 debug-util.cc:398] Leaking SignalData structure 0x7b080035dee0 after lost signal to thread 14141
I20260202 15:08:21.435961 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.425940 14173 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.442919 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.440199 12545 catalog_manager.cc:5654] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 2 to 3, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:21.443432 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.426962 14058 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.421188 14174 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.362947 14193 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.426859 14185 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.429528 14154 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.429025 14152 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.421211 14195 consensus_queue.cc:1048] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260202 15:08:21.428994 12643 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:21.432893 14130 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.423305 14179 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.462072 13362 raft_consensus.cc:1217] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260202 15:08:21.462144 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.462668 14090 raft_consensus.cc:993] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: : Instructing follower bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:21.446633 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.438774 14158 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.446409 14197 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.457530 12642 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:21.454002 12615 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.426417 14151 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.442716 14173 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.447327 14058 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.454300 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.469728 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.469884 14179 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.473801 14040 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.456049 14152 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.476506 12647 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:21.447845 14176 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.447022 14183 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.480278 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.444650 14201 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.481715 14090 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.481941 14199 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:21.484247 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.485800 14040 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.485873 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.482069 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.481132 14058 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.476888 14173 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.487934 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:21.493969 12796 raft_consensus.cc:3060] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.488930 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.489219 12718 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:21.487346 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.442111 14200 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.498273 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.483705 14152 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.499193 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.424047 14205 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.491863 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.492357 12641 raft_consensus.cc:1240] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:21.459133 14185 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:21.495213 12650 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 1.
I20260202 15:08:21.493510 14209 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.495657 14204 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.496268 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.496834 13358 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 2: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:21.491037 14203 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.501556 12650 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:21.500372 12795 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:21.498188 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.471048 14130 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:21.506094 12614 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.479970 14158 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.511809 14040 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.512143 14040 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.512455 14040 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.494303 14199 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.513204 12615 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.514640 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.506960 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:21.498823 12797 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:21.523598 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.507328 14102 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.522089 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.520984 12718 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:21.532076 12794 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.534708 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.472826 14197 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.537117 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.511749 14191 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.504505 14090 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.521111 14199 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:21.539284 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.512859 14204 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.546409 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.512595 14195 consensus_queue.cc:1059] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260202 15:08:21.510414 14171 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260202 15:08:21.556231 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.514102 14203 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.557472 12723 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:21.520874 14211 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.519204 14209 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.570487 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.571254 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.571449 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.571983 12726 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:21.552774 14174 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.553303 14204 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.552175 14187 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.589907 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.557458 14215 raft_consensus.cc:1081] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Signalling peer bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:21.557371 14202 consensus_queue.cc:1059] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20260202 15:08:21.557719 14166 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.593818 14202 raft_consensus.cc:3055] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:21.600950 14202 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:21.567467 14203 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.635330 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.650307 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.637344 14176 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:21.583171 14211 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.644881 14185 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:21.665776 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.638434 14193 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.673552 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.649919 14183 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.660563 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.681587 12796 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:21.655686 14151 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.665135 12646 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:21.683456 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.685436 12643 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:34792
I20260202 15:08:21.685902 12643 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:21.686213 12643 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.686228 14155 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.687892 12688 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.688124 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.664392 14211 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.690016 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.692901 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.699903 14040 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.701426 12648 raft_consensus.cc:1240] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:21.702668 14040 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.633667 14202 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.706866 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.685243 14158 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.692345 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.720927 13354 raft_consensus.cc:1217] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20260202 15:08:21.727207 13360 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:21.721182 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.707926 14183 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.736506 14214 raft_consensus.cc:3055] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:21.736835 14214 raft_consensus.cc:740] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:21.739884 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.742411 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.731258 14130 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.735232 14202 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.739475 14214 consensus_queue.cc:260] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.732861 14155 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.734160 14183 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:21.764569 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 353939us
I20260202 15:08:21.769956 14189 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.778213 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.779946 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.791630 12795 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:21.795357 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.780700 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.781262 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.783993 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.721313 14204 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:21.786404 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 273269us
I20260202 15:08:21.780522 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.752270 14203 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.798499 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:21.803740 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:21.802436 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.735543 14200 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.734997 14209 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.814426 12765 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.791801 14183 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.786814 14205 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.726053 14215 consensus_queue.cc:1059] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260202 15:08:21.816874 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.821023 12795 raft_consensus.cc:1240] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:21.825798 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.786423 14058 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:21.828146 12765 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:21.741653 14197 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.829265 12765 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.752290 14199 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:21.830088 12615 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.802187 14217 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.835562 12765 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.836182 12615 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.817776 14185 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.841682 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.839926 14208 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.842674 14208 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.843103 14208 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.839944 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.822630 14214 raft_consensus.cc:3060] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.842610 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.844480 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.845850 12690 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:21.847460 12800 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:21.848129 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.831638 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.850329 12647 raft_consensus.cc:1240] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:21.849820 12615 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.856029 12690 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.857781 12690 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.830994 14188 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.840045 14113 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.859592 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.861608 12800 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:21.859838 14040 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.832653 14090 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.862128 14040 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.864246 14040 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.862319 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.843673 14168 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.861028 14158 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:21.874316 12643 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.875576 14058 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.831916 14178 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.881675 12794 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:21.847986 14222 raft_consensus.cc:2749] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:21.883190 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:21.876108 12643 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.883459 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.884727 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.885721 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.887523 14209 raft_consensus.cc:3055] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:21.861465 14113 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.888068 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.888392 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:21.888527 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.865003 14150 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.855366 14217 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.889850 12690 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.859089 14215 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.861827 14193 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:21.848628 14220 raft_consensus.cc:2804] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:21.887384 14130 consensus_queue.cc:1059] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:21.848322 14185 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.891525 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.893814 12797 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.897145 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:21.897835 14223 log.cc:927] Time spent T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260202 15:08:21.897856 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.902519 14150 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:21.889492 14168 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
W20260202 15:08:21.899195 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 386724us
I20260202 15:08:21.906194 14040 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.907951 12791 raft_consensus.cc:1240] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:21.920976 14040 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.909420 12615 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.863668 14188 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.899430 14113 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.897833 14189 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:21.894109 14209 raft_consensus.cc:740] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:21.912536 12794 raft_consensus.cc:1275] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:21.919219 12799 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:21.900234 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.909077 14216 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:21.923535 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 88033.3us
I20260202 15:08:21.910413 14215 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.923816 12614 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:21.926753 12690 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.914077 14220 raft_consensus.cc:697] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:21.928771 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.929296 12690 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.930169 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.930110 12765 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.930456 12690 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.930963 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:21.931233 12765 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:21.931511 12690 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:21.917652 14204 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.921754 14168 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.932946 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.932953 12690 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:21.923172 14150 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.935130 14195 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.934254 12765 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:21.934438 14174 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:21.936650 14217 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.932621 14101 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:21.926131 14189 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.918570 12793 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:21.940434 12614 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.926286 14155 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:21.926714 14203 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.929438 14215 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.942309 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.945303 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.949139 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:21.931128 14193 raft_consensus.cc:2749] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:21.945195 12614 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:21.949985 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:21.950194 14040 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.931450 14166 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:21.942371 14199 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:21.950493 12614 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.941867 14221 raft_consensus.cc:2804] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.950528 14040 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.949985 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.943363 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:21.957350 14155 raft_consensus.cc:697] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:21.958680 13326 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:21.963366 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:21.940971 14195 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.960705 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.956445 14040 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.936712 14201 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.950066 14192 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.947681 14151 raft_consensus.cc:2804] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.947789 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:21.951341 14217 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:21.967648 14200 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:21.968145 14205 raft_consensus.cc:2804] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:21.959743 14187 raft_consensus.cc:2764] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260202 15:08:22.017275 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:21.971356 14185 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:21.965935 12643 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.965111 14195 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:21.926029 14209 consensus_queue.cc:260] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:21.957145 14221 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:21.990183 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 48916.1us
I20260202 15:08:21.968890 14192 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:22.020298 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:21.971594 14151 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.975203 14193 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:22.028208 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.981997 14217 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:21.990540 14205 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:21.989838 12794 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.030805 14192 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:21.982160 14178 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.030861 14151 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.034503 14155 consensus_queue.cc:237] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.035132 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:21.981290 14220 consensus_queue.cc:237] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.061683 14090 raft_consensus.cc:3055] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:22.062067 14090 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:22.062883 14090 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:22.048800 14205 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:22.047930 14217 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.026583 14221 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
W20260202 15:08:22.078725 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 53173.3us
I20260202 15:08:21.969609 12690 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.086336 14040 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:22.092695 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 123127us
I20260202 15:08:22.093500 12690 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.098346 14040 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.099500 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:22.100466 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.101130 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.101693 13356 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.101935 12690 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.100443 14186 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.099762 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.104269 14186 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.105402 12690 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.107038 14186 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:22.108068 12690 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.109647 13356 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:22.076725 14209 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:22.111891 12765 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.113287 12690 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:22.114782 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.115027 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.115368 12719 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:22.116143 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.116138 12765 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.083842 14090 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:22.118265 12765 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:22.120240 12765 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.108098 14215 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.120422 12690 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:22.110364 14158 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.110148 14186 raft_consensus.cc:2749] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.119351 14206 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.129172 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:22.119187 14185 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.124094 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.132889 13354 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:22.124043 14178 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:22.138212 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.138330 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:22.122356 14201 raft_consensus.cc:2804] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:22.133009 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:22.143975 14152 consensus_meta.cc:287] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.634s user 0.003s sys 0.018s
I20260202 15:08:22.144296 14215 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.124197 14174 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:22.145527 14174 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:22.145658 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:22.145843 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.123898 14222 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.142477 14188 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.144399 14152 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.144996 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:22.140770 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 163940us
I20260202 15:08:22.147948 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.148154 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:22.148684 12736 tablet_copy_service.cc:342] P 6921687ed3814cccb2bd9c606875b9f0: Request end of tablet copy session 66b8c715b9e6479e9218b08bd3156380-015f643434d84eac8a3774b663eb4220 received from {username='slave'} at 127.0.0.1:59480
W20260202 15:08:22.150014 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:22.119124 14193 consensus_queue.cc:1048] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260202 15:08:22.156342 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:22.160526 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:22.161582 14173 consensus_meta.cc:287] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.652s user 0.006s sys 0.013s
I20260202 15:08:22.161810 14150 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.145902 14174 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.163031 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.173250 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.167817 12614 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:22.142019 14201 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:22.170306 12736 tablet_copy_service.cc:434] P 6921687ed3814cccb2bd9c606875b9f0: ending tablet copy session 66b8c715b9e6479e9218b08bd3156380-015f643434d84eac8a3774b663eb4220 on tablet 015f643434d84eac8a3774b663eb4220 with peer 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:22.163703 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.174089 12796 raft_consensus.cc:1240] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.170053 12791 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.181649 12726 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:22.181689 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.182593 12719 raft_consensus.cc:3055] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:22.179468 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.181502 14154 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:22.168699 12643 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:22.189142 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.204790 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.204512 12725 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:22.170187 14130 consensus_queue.cc:1059] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260202 15:08:22.182888 12719 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:22.209035 14130 raft_consensus.cc:3055] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:22.188885 14154 raft_consensus.cc:697] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:22.209241 12719 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.210340 12719 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.210085 14154 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.181354 14201 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.205536 12792 raft_consensus.cc:2393] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:22.170969 14208 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.174383 14173 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.226353 12688 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:22.227070 14193 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:22.227458 12794 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:22.219123 12645 raft_consensus.cc:1240] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:22.223984 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.230187 12723 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.231144 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.209355 14130 raft_consensus.cc:740] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:22.229041 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.233665 12615 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:22.253175 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:22.230283 14150 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:22.217646 12797 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:22.262584 12795 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:22.286031 12800 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:22.227490 14193 raft_consensus.cc:697] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:22.285432 14233 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:22.285663 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:22.237737 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:22.293984 12643 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:22.308266 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:22.308493 13328 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.309248 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:22.310165 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:22.310292 13328 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:22.315690 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 86235.2us
I20260202 15:08:22.317006 12615 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:22.316644 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.317807 12615 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:22.319636 13328 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.319409 12765 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.318493 14231 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.320706 12643 raft_consensus.cc:1240] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.285972 14214 consensus_queue.cc:1059] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260202 15:08:22.323194 14232 raft_consensus.cc:2749] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.317098 12721 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:22.312377 14233 raft_consensus.cc:697] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:22.326835 12796 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.321767 14058 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:22.327888 12763 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.328794 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.330042 12796 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.329777 14193 consensus_queue.cc:237] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.330225 14058 raft_consensus.cc:697] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:22.329360 12763 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.318881 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:22.328488 14188 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.335625 12763 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.332409 14101 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:22.338406 14230 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.338084 14233 consensus_queue.cc:237] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.334411 14200 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.345675 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:22.341825 12763 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.346580 12796 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.325709 14152 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.344740 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:22.347484 12763 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.349310 12763 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.351120 12763 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:22.353741 13328 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.355135 13328 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.339432 14190 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.318951 14130 consensus_queue.cc:260] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.344833 14222 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:22.342825 14208 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.351764 14232 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:22.349061 14188 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:22.334793 14102 consensus_queue.cc:1059] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260202 15:08:22.369038 12646 raft_consensus.cc:1240] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.348517 14150 raft_consensus.cc:2764] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260202 15:08:22.348450 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.371229 12798 raft_consensus.cc:1240] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.360435 14101 raft_consensus.cc:2764] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260202 15:08:22.342821 14195 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.339380 14151 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.382061 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:22.369156 14202 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:22.348285 14189 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.361403 14231 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:22.387537 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:22.336226 14058 consensus_queue.cc:237] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.396435 12718 raft_consensus.cc:1240] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.384451 14130 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:22.376639 14173 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.390827 12642 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:22.389647 14213 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:22.382081 14215 consensus_queue.cc:1059] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260202 15:08:22.409320 12615 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.373135 14179 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260202 15:08:22.379323 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.372959 14197 consensus_queue.cc:1059] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260202 15:08:22.388674 14188 consensus_queue.cc:1059] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260202 15:08:22.396162 14195 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:22.395664 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:22.436017 12648 raft_consensus.cc:1240] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.401736 12791 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:22.404120 14205 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.389364 12763 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.404650 14151 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.390036 13356 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.411983 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:22.449522 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 60226.7us
I20260202 15:08:22.450230 12763 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.450318 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.417903 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:22.430343 13328 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.389216 14206 consensus_queue.cc:1059] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260202 15:08:22.398412 14183 consensus_meta.cc:287] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.571s user 0.021s sys 0.044s
I20260202 15:08:22.395342 12795 raft_consensus.cc:1240] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:22.400231 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.436455 12615 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:22.443145 12796 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.453714 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.389822 14232 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.456948 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.457619 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.455720 12690 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:22.445806 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:22.447418 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:22.451830 13328 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.435631 12792 raft_consensus.cc:1240] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.438410 14199 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:22.442281 12793 raft_consensus.cc:1240] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.463196 13360 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:22.463797 13328 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.466027 12763 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.462525 12615 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:22.445492 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:22.469470 12615 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.468181 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:22.456650 14222 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:22.463621 14220 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:22.475646 12615 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.465124 14179 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.464938 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:22.487042 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:22.467842 14150 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:22.491230 12722 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:22.496091 12689 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:22.498392 14179 raft_consensus.cc:2749] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.437764 14231 consensus_queue.cc:1059] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260202 15:08:22.475122 14215 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:22.460892 14183 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:22.492568 14211 consensus_meta.cc:287] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.777s user 0.009s sys 0.000s
I20260202 15:08:22.476998 14154 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.483331 14197 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.491050 14220 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:22.492815 14222 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.519658 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.500169 14153 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:22.525738 12724 raft_consensus.cc:1240] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.504455 14158 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:22.524186 14239 log.cc:927] Time spent T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260202 15:08:22.521468 14220 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:22.496644 14188 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260202 15:08:22.515895 14215 raft_consensus.cc:697] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:22.535760 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:22.537731 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.538470 12545 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45944 (request call id 308) took 1128 ms. Trace:
I20260202 15:08:22.538763 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.539345 12726 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.477895 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:22.530232 14153 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.551267 14153 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:22.551879 14153 raft_consensus.cc:1064] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:22.530284 14231 consensus_queue.cc:1243] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:08:22.518306 14211 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.559216 14210 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:22.559669 14210 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:22.559991 14210 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.559294 14211 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.502640 14155 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:22.547834 14189 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:22.577668 14168 consensus_meta.cc:287] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.634s user 0.014s sys 0.013s
I20260202 15:08:22.577986 14168 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.582418 14204 raft_consensus.cc:3055] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:22.582480 14183 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.582894 14204 raft_consensus.cc:740] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:22.573487 13356 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:22.583667 14204 consensus_queue.cc:260] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.584240 13359 raft_consensus.cc:1240] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: []
W20260202 15:08:22.585711 14192 consensus_meta.cc:287] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.534s user 0.018s sys 0.004s
I20260202 15:08:22.586117 14192 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:22.591446 14240 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260202 15:08:22.585728 12764 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.592474 14188 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:22.592964 14188 raft_consensus.cc:697] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:22.593799 14188 consensus_queue.cc:237] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.594748 14173 raft_consensus.cc:3055] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:22.575700 14205 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:22.571655 14217 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.612690 14215 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.613745 14178 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
W20260202 15:08:22.616195 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:22.619638 12725 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:22.620425 14227 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260202 15:08:22.598344 14173 raft_consensus.cc:740] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:22.631524 12795 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:08:22.627326 14113 consensus_meta.cc:287] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.698s user 0.011s sys 0.014s
I20260202 15:08:22.622419 12649 raft_consensus.cc:1240] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
W20260202 15:08:22.629160 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 157940us
I20260202 15:08:22.538918 12545 rpcz_store.cc:276] 0202 15:08:21.409623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:21.409731 (+ 108us) service_pool.cc:225] Handling call
0202 15:08:22.538380 (+1128649us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:08:21.862762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:08:21.873650 (+ 10888us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:08:21.873706 (+ 56us) write_op.cc:435] Acquired schema lock
0202 15:08:21.873730 (+ 24us) tablet.cc:660] Decoding operations
0202 15:08:21.874161 (+ 431us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:08:21.874262 (+ 101us) write_op.cc:641] Partition lock acquired for write op
0202 15:08:21.874312 (+ 50us) tablet.cc:683] Acquiring locks for 1 operations
0202 15:08:21.874544 (+ 232us) tablet.cc:699] Row locks acquired
0202 15:08:21.874572 (+ 28us) write_op.cc:260] PREPARE: finished
0202 15:08:21.874759 (+ 187us) write_op.cc:270] Start()
0202 15:08:21.874928 (+ 169us) write_op.cc:276] Timestamp: P: 1770044901874690 usec, L: 0
0202 15:08:21.874968 (+ 40us) op_driver.cc:348] REPLICATION: starting
0202 15:08:21.875686 (+ 718us) log.cc:844] Serialized 536 byte log entry
0202 15:08:22.098284 (+222598us) op_driver.cc:464] REPLICATION: finished
0202 15:08:22.288081 (+189797us) write_op.cc:301] APPLY: starting
0202 15:08:22.294608 (+ 6527us) tablet.cc:1365] starting BulkCheckPresence
0202 15:08:22.311683 (+ 17075us) tablet.cc:1368] finished BulkCheckPresence
0202 15:08:22.328641 (+ 16958us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:08:22.339139 (+ 10498us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:08:22.339282 (+ 143us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0202 15:08:22.339330 (+ 48us) write_op.cc:312] APPLY: finished
0202 15:08:22.357319 (+ 17989us) log.cc:844] Serialized 23 byte log entry
0202 15:08:22.357490 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:22.357564 (+ 74us) write_op.cc:454] Released schema lock
0202 15:08:22.357601 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0202 15:08:22.357950 (+ 349us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:22.357986 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":124513,"apply.run_cpu_time_us":2054,"apply.run_wall_time_us":91432,"num_ops":1,"prepare.queue_time_us":387018,"prepare.run_cpu_time_us":14881,"prepare.run_wall_time_us":73519,"raft.queue_time_us":142446,"raft.run_cpu_time_us":23348,"raft.run_wall_time_us":74531,"replication_time_us":223171,"thread_start_us":620241,"threads_started":4,"wal-append.queue_time_us":64247}]]}
I20260202 15:08:22.614138 14232 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.622689 14178 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.658763 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:22.658735 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:22.681154 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 179954us
I20260202 15:08:22.682232 12689 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.684741 12717 raft_consensus.cc:1240] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.685171 14217 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:22.686980 14221 consensus_meta.cc:287] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.605s user 0.004s sys 0.019s
I20260202 15:08:22.687314 14221 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.688162 12641 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:22.688946 14221 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.663285 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.691164 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.691608 12642 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.692075 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.692209 12719 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:22.693018 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:22.693670 12724 raft_consensus.cc:2393] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
W20260202 15:08:22.698788 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 235052us
I20260202 15:08:22.699558 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.647711 14113 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.705862 12689 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:22.707727 12689 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.708724 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:22.716645 12726 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:22.719092 12724 raft_consensus.cc:1240] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.748468 12799 raft_consensus.cc:1240] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:22.749795 13328 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.747042 14205 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.751269 14248 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:22.714049 14193 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:22.738009 14174 consensus_meta.cc:287] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.565s user 0.007s sys 0.007s
I20260202 15:08:22.766265 12800 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.707837 14192 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.766384 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:22.705971 14231 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260202 15:08:22.667515 14173 consensus_queue.cc:260] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.774398 14229 raft_consensus.cc:1064] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:22.774622 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:22.743403 14168 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:22.795739 12643 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.789408 14243 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.767175 14195 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:22.766666 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:22.822098 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 46684.9us
I20260202 15:08:22.764329 14197 consensus_queue.cc:1059] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260202 15:08:22.843376 14197 raft_consensus.cc:3055] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:22.843801 14197 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:22.844779 14197 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.846251 14197 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.829780 14209 raft_consensus.cc:3055] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:22.846853 14209 raft_consensus.cc:740] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:22.847780 14209 consensus_queue.cc:260] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.850878 14199 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.853629 14113 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:22.772992 14205 consensus_queue.cc:1059] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260202 15:08:22.855376 14205 raft_consensus.cc:3055] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:22.855742 14205 raft_consensus.cc:740] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:22.856611 14205 consensus_queue.cc:260] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.857856 14205 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.859176 14202 raft_consensus.cc:3055] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:22.770145 14174 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.859491 14202 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:22.860333 14176 raft_consensus.cc:3055] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:22.860685 14176 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:22.861126 14174 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.861488 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.861886 12720 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.863205 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.863831 14101 raft_consensus.cc:3055] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:22.864234 14101 raft_consensus.cc:740] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:22.830768 14190 consensus_queue.cc:1059] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260202 15:08:22.831100 14153 consensus_queue.cc:237] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.877326 14173 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:22.835508 14171 consensus_queue.cc:1059] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260202 15:08:22.838514 14201 consensus_meta.cc:287] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.613s user 0.008s sys 0.001s
I20260202 15:08:22.866650 14225 raft_consensus.cc:3055] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:22.867172 14251 consensus_queue.cc:1059] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260202 15:08:22.828244 14230 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:22.819764 14210 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.881201 14190 raft_consensus.cc:3055] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:22.829361 14231 consensus_queue.cc:1243] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260202 15:08:22.913417 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 410509us
I20260202 15:08:22.873879 14178 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.914762 14220 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.917629 12641 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
W20260202 15:08:22.919562 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 154441us
W20260202 15:08:22.920383 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:22.921337 12642 raft_consensus.cc:1240] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.922924 13328 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:22.927007 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1615) took 1065 ms. Trace:
I20260202 15:08:22.903725 14225 raft_consensus.cc:740] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:22.906260 14190 raft_consensus.cc:740] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:22.927349 12799 rpcz_store.cc:276] 0202 15:08:21.861817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:21.974822 (+113005us) service_pool.cc:225] Handling call
0202 15:08:22.019725 (+ 44903us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:22.748981 (+729256us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:22.926801 (+177820us) spinlock_profiling.cc:243] Waited 723 ms on lock 0x7b5400357ad4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:22.926955 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1662875392}
I20260202 15:08:22.865115 14101 consensus_queue.cc:260] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.915882 12615 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.949059 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.951931 12615 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.954248 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:22.957062 12792 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.957389 12717 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:22.884912 14201 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.957312 14225 consensus_queue.cc:260] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:22.972393 12765 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:22.972513 12644 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:22.973518 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:22.974893 12765 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.975112 12641 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:22.960230 12720 raft_consensus.cc:1240] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.977501 14252 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:22.977691 14213 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:22.978199 14213 raft_consensus.cc:697] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:22.978569 14201 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.979604 12723 raft_consensus.cc:1240] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:22.940440 14200 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.982913 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:22.905227 14230 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.960244 14225 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:22.986752 12765 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:22.987771 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:22.987834 12615 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.978941 14213 consensus_queue.cc:237] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:22.954902 14171 consensus_queue.cc:1059] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260202 15:08:22.978600 14178 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:22.971449 14101 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:22.998267 14198 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:22.973320 14179 raft_consensus.cc:3055] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:22.992352 12615 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.971871 14199 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:22.961133 14113 consensus_queue.cc:1059] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260202 15:08:22.971424 14249 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.002576 12642 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:23.003170 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:23.004629 12724 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:23.006945 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:23.007563 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 48655.8us
I20260202 15:08:23.022130 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:23.001513 12792 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:23.015170 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:23.022225 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:23.000551 14178 raft_consensus.cc:2749] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.000411 14179 raft_consensus.cc:740] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:23.023344 12763 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.025310 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:22.941906 14243 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260202 15:08:23.029038 14251 consensus_queue.cc:1059] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:23.036026 12764 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:22.991142 14220 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:23.009451 14233 raft_consensus.cc:2749] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260202 15:08:23.042317 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 334632us
I20260202 15:08:23.032421 12792 raft_consensus.cc:606] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Received request to transfer leadership to TS bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:23.091017 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 53313.5us
I20260202 15:08:23.098083 12689 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:23.029855 14255 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.021786 14200 consensus_queue.cc:1059] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
W20260202 15:08:23.103960 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 85516.6us
I20260202 15:08:23.023924 14199 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:23.107589 12689 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.040591 14178 raft_consensus.cc:2749] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:23.113008 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:23.113811 14199 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:23.090380 14113 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.090718 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:23.099157 14210 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:23.100543 14173 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:23.123291 12615 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:23.109597 14214 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:23.138939 12689 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:23.125579 14247 log.cc:927] Time spent T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
I20260202 15:08:23.096293 14232 consensus_queue.cc:1059] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260202 15:08:23.151955 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:23.153277 12689 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.138136 14198 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:23.136263 14250 consensus_queue.cc:1059] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20260202 15:08:23.118211 14195 consensus_queue.cc:1059] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20260202 15:08:23.153605 12797 raft_consensus.cc:2393] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:23.123329 14173 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:23.138451 14155 consensus_queue.cc:1059] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260202 15:08:23.153715 14130 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.161274 14186 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.115958 14179 consensus_queue.cc:260] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.180471 12689 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.184696 12642 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:23.152712 14258 raft_consensus.cc:2749] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:23.172123 14230 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:23.186381 14195 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:23.187894 14130 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.200157 14173 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:23.190006 14259 consensus_queue.cc:1059] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260202 15:08:23.203331 14259 raft_consensus.cc:3055] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:23.203619 14259 raft_consensus.cc:740] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:23.185844 12689 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.172401 14166 consensus_queue.cc:1059] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260202 15:08:23.145448 14260 consensus_queue.cc:1059] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
W20260202 15:08:23.181690 12615 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.46.194:44367 timed out after 1.531s (SENT) [suppressed 4 similar messages]
I20260202 15:08:23.188933 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:23.226591 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:23.227537 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:23.228600 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:23.228632 12546 catalog_manager.cc:5654] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 2 to 3, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 3 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:23.229734 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:23.230439 12797 raft_consensus.cc:2393] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:23.231367 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:23.232613 12795 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260202 15:08:23.215788 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:23.236301 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 51474.3us
I20260202 15:08:23.238216 12690 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:23.192692 13355 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:23.204422 14259 consensus_queue.cc:260] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.243134 12797 raft_consensus.cc:1275] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:23.241382 14216 raft_consensus.cc:2749] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.235077 14260 raft_consensus.cc:3055] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:23.240442 14214 consensus_queue.cc:1059] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260202 15:08:23.208499 14231 raft_consensus.cc:3055] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:23.264420 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 186) took 1114 ms. Trace:
I20260202 15:08:23.264832 12723 rpcz_store.cc:276] 0202 15:08:22.149919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:22.231844 (+ 81925us) service_pool.cc:225] Handling call
0202 15:08:22.232091 (+ 247us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:22.980200 (+748109us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:23.264149 (+283949us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54003acad4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:23.264347 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1718721792}
I20260202 15:08:23.267629 12795 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
W20260202 15:08:23.297482 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:23.297534 12646 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:23.297857 14268 raft_consensus.cc:3055] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:23.298288 14268 raft_consensus.cc:740] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:23.296378 12688 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.250173 12794 raft_consensus.cc:1240] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:23.250252 14260 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:23.250877 14231 raft_consensus.cc:740] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:08:23.305054 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 118024us
I20260202 15:08:23.302181 14259 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
W20260202 15:08:23.313716 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 132034us
I20260202 15:08:23.302874 14186 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:23.317721 12644 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:23.319908 14267 consensus_queue.cc:1059] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260202 15:08:23.305301 14173 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:23.329734 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 144596us
I20260202 15:08:23.331597 12615 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:23.303066 14250 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260202 15:08:23.333913 12615 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:23.299178 14268 consensus_queue.cc:260] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:23.342265 13355 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:23.342587 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:23.343185 14250 consensus_queue.cc:1243] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:08:23.343458 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:23.344548 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:23.345495 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:23.333706 12646 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:23.346942 12718 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:23.333904 14155 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:23.328949 14231 consensus_queue.cc:260] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.349534 14155 raft_consensus.cc:697] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:23.349809 14231 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:23.350358 14155 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:23.350988 14198 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:23.352366 14198 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:23.336468 14153 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.355602 12615 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:23.356494 14191 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.357297 14153 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:23.343201 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:23.358583 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:23.339990 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:23.341614 13328 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:23.361128 14250 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:23.362905 13328 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.362253 14250 consensus_queue.cc:1059] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:23.362607 13357 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:23.364081 14250 raft_consensus.cc:3055] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:23.364449 14250 raft_consensus.cc:740] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:23.365688 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:23.363309 12689 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.367071 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:23.367575 12718 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:23.366415 13328 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:23.318243 14249 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260202 15:08:23.368462 14267 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:23.369686 13328 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.340353 14230 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:23.340225 14205 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:23.315019 14260 consensus_queue.cc:260] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.371295 14267 raft_consensus.cc:2749] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.370600 12615 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:23.385948 12615 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:23.369625 14173 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.367287 12797 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:23.365895 14243 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:23.386368 14205 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:23.396521 14197 consensus_meta.cc:287] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.550s user 0.008s sys 0.003s
I20260202 15:08:23.396598 14205 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:23.385022 14268 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:23.397046 12718 raft_consensus.cc:1275] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:23.360106 14152 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260202 15:08:23.389308 14210 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.405550 14252 raft_consensus.cc:3055] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:23.405947 14252 raft_consensus.cc:740] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:23.382273 14230 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.382494 14249 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260202 15:08:23.384636 14200 consensus_queue.cc:1059] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260202 15:08:23.391726 14267 consensus_queue.cc:1059] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260202 15:08:23.410894 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:23.413293 12721 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:23.408011 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:23.416512 12646 raft_consensus.cc:2393] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:23.417675 12797 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:23.418893 12797 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:23.419832 12797 raft_consensus.cc:1240] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:23.392436 14101 consensus_queue.cc:1059] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260202 15:08:23.406687 14252 consensus_queue.cc:260] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:23.426188 14254 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:23.428887 12718 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:23.365232 14250 consensus_queue.cc:260] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.450556 12642 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:23.448259 12615 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:23.408526 14192 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260202 15:08:23.445017 14205 raft_consensus.cc:3055] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
W20260202 15:08:23.467919 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 85696.5us
I20260202 15:08:23.448007 13355 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:23.472626 14233 raft_consensus.cc:3055] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:23.466506 14250 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:23.447185 14220 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260202 15:08:23.452958 14173 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260202 15:08:23.489564 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 188) took 1518 ms. Trace:
W20260202 15:08:23.457005 14251 consensus_queue.cc:1175] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "bbf140958c3b4f339c512a9dc4be2dab" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20260202 15:08:23.466285 14205 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:23.460726 14200 consensus_queue.cc:1059] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260202 15:08:23.498498 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:23.497072 13328 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:23.505206 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:23.476457 14233 raft_consensus.cc:740] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:23.511689 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:23.510444 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:23.513470 12718 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:23.515254 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:23.515902 12718 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:23.516897 13328 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:23.517962 13328 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:23.479638 14231 consensus_queue.cc:1059] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260202 15:08:23.519909 14200 raft_consensus.cc:2749] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:23.531503 12545 catalog_manager.cc:5654] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 2 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:23.541450 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:23.571352 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:23.494963 12794 rpcz_store.cc:276] 0202 15:08:21.954662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:22.315013 (+360351us) service_pool.cc:225] Handling call
0202 15:08:22.803388 (+488375us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:23.240567 (+437179us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:23.303868 (+ 63301us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:23.473263 (+169395us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b5400117450. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:23.473372 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":999720064}
I20260202 15:08:23.536243 14230 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:23.597929 12689 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:23.600410 12689 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:23.540787 14205 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.597048 14101 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:23.598989 14102 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:23.602802 14186 raft_consensus.cc:2804] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:23.611007 14276 consensus_queue.cc:1059] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260202 15:08:23.603619 14230 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:23.602562 14258 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:23.605026 14200 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:23.619745 12794 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:23.547334 14233 consensus_queue.cc:260] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:23.617690 14101 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:23.620033 14202 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.632229 14230 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:23.640287 14200 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:23.649230 14230 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:23.673770 12799 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:23.618041 14102 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:23.669754 14200 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:23.648737 14090 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260202 15:08:23.671057 14233 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:23.677598 12726 raft_consensus.cc:1275] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:23.673111 14186 raft_consensus.cc:697] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:23.682958 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:23.707152 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:23.736245 14286 log.cc:927] Time spent T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260202 15:08:23.806365 14215 consensus_queue.cc:1059] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260202 15:08:23.774341 14277 log.cc:927] Time spent T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20260202 15:08:23.871320 12717 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:23.780309 12721 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:23.917114 12647 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:23.932199 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:23.941183 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:23.873571 14101 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:23.946534 14186 consensus_queue.cc:237] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.959422 14230 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.974084 14284 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:23.974215 14248 raft_consensus.cc:3055] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:23.974738 14248 raft_consensus.cc:740] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:23.974671 14284 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.971266 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:23.978542 14230 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:23.977020 14248 consensus_queue.cc:260] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:23.971800 14101 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260202 15:08:23.995966 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 411) took 1365 ms. Trace:
I20260202 15:08:24.001154 14200 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:23.996605 13354 rpcz_store.cc:276] 0202 15:08:22.630489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:22.640123 (+ 9634us) service_pool.cc:225] Handling call
0202 15:08:22.988471 (+348348us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:22.988632 (+ 161us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:23.028439 (+ 39807us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:23.032465 (+ 4026us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:23.117774 (+ 85309us) log.cc:844] Serialized 26 byte log entry
0202 15:08:23.296091 (+178317us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:23.301568 (+ 5477us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:23.995518 (+693950us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b5400511654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:23.995592 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:23.995659 (+ 67us) raft_consensus.cc:1685] finished
0202 15:08:23.995685 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:23.995859 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16006,"raft.queue_time_us":21529,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"spinlock_wait_cycles":32453760,"thread_start_us":107260,"threads_started":1,"wal-append.queue_time_us":181217}
I20260202 15:08:24.003582 14200 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.005739 13354 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:24.008309 12793 raft_consensus.cc:1275] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:24.010972 14279 raft_consensus.cc:3055] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:24.011350 14279 raft_consensus.cc:740] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:24.011466 14284 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:24.013262 13357 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:23.988960 14102 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.015743 14279 consensus_queue.cc:260] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.018262 14279 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:24.042124 13328 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.001s sys 0.036s
W20260202 15:08:24.043025 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:24.044521 13328 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s user 0.002s sys 0.036s
W20260202 15:08:24.048738 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 530809us
I20260202 15:08:24.062891 14150 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.063401 14150 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.064834 14190 consensus_queue.cc:260] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.070420 12722 raft_consensus.cc:1240] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:24.071466 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:24.072408 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:24.072717 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:24.072978 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.047802 14130 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.065393 14150 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:24.051532 14152 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:24.084116 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.085505 13328 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:24.067768 14206 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:24.091325 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.093164 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:24.093426 12722 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:24.077396 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35714 (request call id 12) took 1100 ms. Trace:
I20260202 15:08:24.102442 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.090839 14190 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:24.088251 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.106328 13328 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:24.086452 14130 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:24.087852 14152 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.109827 14101 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.113538 14282 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:24.094439 14206 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.119810 14291 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.121655 14282 raft_consensus.cc:697] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:24.118985 14204 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:24.119597 14229 consensus_queue.cc:237] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.095432 12792 rpcz_store.cc:276] 0202 15:08:22.962240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:22.972135 (+ 9895us) service_pool.cc:225] Handling call
0202 15:08:23.343108 (+370973us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b54001f5520. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4dc550 00007f24ed5592d2 00007f24f251713a 00007f24ebab2934 00007f24ebab28c2 00007f24ebab27f1 00007f24ebab276c 00007f24ebab1a22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:24.024351 (+681243us) spinlock_profiling.cc:243] Waited 38.7 ms on lock 0x7b54001f5520. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4da617 00007f24ed5598e8 00007f24ed5594ca 00007f24f251713a 00007f24ebab2934 00007f24ebab28c2 00007f24ebab27f1 00007f24ebab276c 00007f24ebab1a22 00007f24eb89e34c
0202 15:08:24.059231 (+ 34880us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":144307456}
I20260202 15:08:24.119669 14101 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:24.127142 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 678891us
I20260202 15:08:24.127322 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:24.129439 12726 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:24.134642 13328 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:24.135540 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.135579 14284 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
W20260202 15:08:24.136129 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.136140 14284 raft_consensus.cc:697] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:24.136698 12649 raft_consensus.cc:1240] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:24.137091 14284 consensus_queue.cc:237] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.138172 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1309) took 1174 ms. Trace:
I20260202 15:08:24.139008 14252 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:24.152345 12725 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:24.121719 14291 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.168412 14202 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:24.172137 14130 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.173641 14152 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:24.136140 13359 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:24.140564 14198 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:24.176376 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.176671 12615 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:24.205494 14291 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.158272 14261 consensus_queue.cc:1059] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20260202 15:08:24.204974 14154 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:24.220014 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.220080 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.125968 14282 consensus_queue.cc:237] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.220594 12642 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:24.138476 12649 rpcz_store.cc:276] 0202 15:08:22.963573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:22.973143 (+ 9570us) service_pool.cc:225] Handling call
0202 15:08:23.931060 (+957917us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:24.132914 (+201854us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:24.137050 (+ 4136us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:24.137963 (+ 913us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54000f96d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:24.138108 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":431033344}
I20260202 15:08:24.224534 12645 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:24.225812 12644 raft_consensus.cc:1240] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:24.187445 14285 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:24.222436 14158 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.802s user 0.017s sys 0.057s
I20260202 15:08:24.207319 14155 raft_consensus.cc:2749] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:24.235244 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.249920 14291 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:24.250355 14291 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.251529 12792 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
W20260202 15:08:24.251694 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 139101us
I20260202 15:08:24.251945 14291 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:24.252502 14208 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:24.252830 12791 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:24.252920 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:24.252992 14208 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:24.254009 12793 raft_consensus.cc:2393] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:24.255272 14208 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:24.255848 14208 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:24.256176 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.256395 14208 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.257689 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.258316 12718 raft_consensus.cc:2393] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:24.258265 14208 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:24.259606 14152 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:24.260178 14300 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:24.260033 14152 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.260576 14300 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.261790 14152 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.262415 14300 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.262979 14152 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.265969 14152 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.267711 14152 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.271347 14152 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:24.264055 12718 raft_consensus.cc:1240] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:24.273150 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:24.263350 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:24.274039 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.273916 12797 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:24.272552 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:24.265159 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.223930 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 420) took 1030 ms. Trace:
W20260202 15:08:24.237452 14158 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.812s user 0.017s sys 0.057s
I20260202 15:08:24.225672 14154 consensus_queue.cc:1048] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
W20260202 15:08:24.282325 14294 log.cc:927] Time spent T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20260202 15:08:24.281311 14200 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.238986 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.283339 14200 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.285594 14200 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.282522 14155 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:24.286902 14155 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.293485 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:24.275606 12794 raft_consensus.cc:1240] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:24.298682 12690 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:24.233273 14168 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260202 15:08:24.279515 14208 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:24.307721 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1639) took 1288 ms. Trace:
I20260202 15:08:24.279630 14299 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:24.303272 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.309737 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.310402 12647 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:24.308099 12798 rpcz_store.cc:276] 0202 15:08:23.019052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.035327 (+ 16275us) service_pool.cc:225] Handling call
0202 15:08:24.307185 (+1271858us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:24.307338 (+ 153us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:24.307479 (+ 141us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:24.307499 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:24.307530 (+ 31us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:24.307554 (+ 24us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:24.307603 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:24.307678 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2709760}
I20260202 15:08:24.314090 12690 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:24.239845 14285 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.314973 12763 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:24.315147 12614 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:24.286919 14265 raft_consensus.cc:3055] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:24.310237 14206 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:24.278115 13359 rpcz_store.cc:276] 0202 15:08:23.191457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.191994 (+ 537us) service_pool.cc:225] Handling call
0202 15:08:24.135860 (+943866us) spinlock_profiling.cc:243] Waited 1.79 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea47412e 000000000035a3c9 00007f24f249e560 00007f24f2566dd4 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:24.135982 (+ 122us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:24.185474 (+ 49492us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:24.221644 (+ 36170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4120832}
I20260202 15:08:24.294782 14150 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.301877 12643 raft_consensus.cc:1240] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:24.309877 14171 raft_consensus.cc:3055] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:24.288527 14243 consensus_queue.cc:1059] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260202 15:08:24.331194 12763 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:24.353724 12718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:24.358345 13362 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:24.322016 14265 raft_consensus.cc:740] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:24.359575 12718 raft_consensus.cc:606] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:24.328050 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:24.373876 12718 raft_consensus.cc:614] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260202 15:08:24.359894 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 154088us
I20260202 15:08:24.334578 14171 raft_consensus.cc:740] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:24.358660 14190 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:24.333292 14186 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:24.375398 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 44282.8us
I20260202 15:08:24.329952 14214 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:24.384162 12647 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:24.384341 12648 raft_consensus.cc:1240] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:24.385322 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.385776 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:24.386556 12642 raft_consensus.cc:2393] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:24.386798 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.387027 13326 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:24.386696 14265 consensus_queue.cc:260] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.387667 12643 raft_consensus.cc:2393] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:24.387758 14200 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:24.388110 14265 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:24.388437 14200 consensus_queue.cc:1243] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:08:24.388923 14200 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:24.390851 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260202 15:08:24.391383 13326 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:24.392390 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:24.325263 14208 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.361406 14250 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.352181 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:24.350786 14155 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.333192 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.328977 14101 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:24.402201 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 101386us
I20260202 15:08:24.361523 14301 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:24.333556 14150 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.404623 14301 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:24.405025 14301 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:24.360165 14303 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:24.405797 12641 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:24.406123 14150 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.396871 14202 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:24.374547 14190 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.378944 14214 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:24.411732 14214 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:24.313529 14299 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.395597 14291 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:24.414350 14299 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:24.417543 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.417802 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.400789 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:24.423548 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.418830 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1528) took 1078 ms. Trace:
I20260202 15:08:24.427377 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.428406 12718 raft_consensus.cc:2393] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 2: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:24.418278 14198 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:24.401846 14252 consensus_queue.cc:1048] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260202 15:08:24.402261 14250 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.434363 14208 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.436261 14304 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.422564 14285 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:24.382802 14171 consensus_queue.cc:260] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.446810 14171 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:24.447144 14190 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.448519 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.448956 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:24.450795 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.443689 12800 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:24.434538 14202 raft_consensus.cc:2749] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:24.427392 12717 rpcz_store.cc:276] 0202 15:08:23.336044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.336509 (+ 465us) service_pool.cc:225] Handling call
0202 15:08:24.401725 (+1065216us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b54002ff154. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:24.414804 (+ 13079us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":788881280}
I20260202 15:08:24.437496 14173 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.456009 14285 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:24.458372 14173 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.458632 14285 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.443020 14243 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:24.459149 14214 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:24.456315 14304 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.459617 14243 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.465572 14214 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.465727 14243 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.466746 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:24.420500 14176 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.468384 14176 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:24.471873 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:24.464179 14197 raft_consensus.cc:3055] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:24.475512 14197 raft_consensus.cc:740] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:24.476207 14197 consensus_queue.cc:260] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.477182 14197 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:24.478235 14260 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:24.449657 12719 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:24.481578 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1312) took 1440 ms. Trace:
I20260202 15:08:24.483242 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 617) took 1151 ms. Trace:
I20260202 15:08:24.483619 13363 rpcz_store.cc:276] 0202 15:08:23.331434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.331553 (+ 119us) service_pool.cc:225] Handling call
0202 15:08:24.482527 (+1150974us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:24.482737 (+ 210us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:24.482922 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:24.482952 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:24.482996 (+ 44us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:24.483031 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:24.483090 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:24.483187 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3396992}
I20260202 15:08:24.481909 12795 rpcz_store.cc:276] 0202 15:08:23.040992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.239786 (+198794us) service_pool.cc:225] Handling call
0202 15:08:23.267401 (+ 27615us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:23.296785 (+ 29384us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:24.481414 (+1184629us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b540036e550. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:24.481525 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39971584}
I20260202 15:08:24.490919 12765 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:24.490232 14198 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:24.491698 14213 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:24.492641 14213 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:24.492947 14304 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.481138 14252 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.492110 14285 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.492530 14198 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.497599 14243 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:24.495172 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.493075 14213 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:24.510823 14301 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.511806 12722 raft_consensus.cc:1240] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:24.514724 14252 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:24.516778 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.513136 14214 raft_consensus.cc:1064] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:24.513514 14202 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.489094 14287 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:24.553042 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.517709 14243 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.581334 14179 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:24.544178 14202 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.578784 14287 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.667794 13360 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260202 15:08:24.669191 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:24.671969 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.678902 14213 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.680706 14213 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.681478 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.682107 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:24.682857 14198 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:24.684711 14202 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.684842 14205 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:24.686420 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:24.689132 12800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1637) took 1675 ms (client timeout 2104 ms). Trace:
W20260202 15:08:24.689482 12800 rpcz_store.cc:269] 0202 15:08:23.013562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.014890 (+ 1328us) service_pool.cc:225] Handling call
0202 15:08:24.688998 (+1674108us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54001f81d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:24.689071 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3014683392}
I20260202 15:08:24.690953 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:24.691180 14209 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:24.691448 12722 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:24.708385 12722 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:24.709052 12725 raft_consensus.cc:1240] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
W20260202 15:08:24.728312 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 327051us
I20260202 15:08:24.732075 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:24.732630 13362 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
W20260202 15:08:24.733794 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 355576us
I20260202 15:08:24.735013 13362 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:24.728259 12765 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:24.737860 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:24.743707 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:24.744300 13363 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:24.709818 14265 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.750283 13363 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Advancing to term 4
I20260202 15:08:24.761866 14246 raft_consensus.cc:1027] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: attempt to promote peer 66b8c715b9e6479e9218b08bd3156380: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260202 15:08:24.722795 14213 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:24.766316 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.766666 12763 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:24.768525 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.726389 14301 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:24.772524 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 370792us
I20260202 15:08:24.775290 12615 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:24.776432 12763 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:24.782627 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.779032 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.784462 12642 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:24.763947 14252 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:24.792290 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.769356 14183 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:24.810187 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 431055us
I20260202 15:08:24.829701 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:24.806906 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:24.806082 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:24.837373 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.839036 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.765599 14216 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260202 15:08:24.840910 12645 raft_consensus.cc:2393] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:24.739423 14287 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.831965 12798 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
W20260202 15:08:24.841840 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 77042.8us
I20260202 15:08:24.852675 12614 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:24.804605 14206 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:24.788259 14282 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.804315 14202 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.776975 14213 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.854187 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 408) took 1735 ms. Trace:
I20260202 15:08:24.804459 14208 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:24.857384 13328 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:24.763722 14265 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.864061 13360 rpcz_store.cc:276] 0202 15:08:23.119534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.119635 (+ 101us) service_pool.cc:225] Handling call
0202 15:08:23.119865 (+ 230us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:24.668200 (+1548335us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:24.853911 (+185711us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400512054. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:24.854120 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3559117696}
W20260202 15:08:24.861410 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 482258us
I20260202 15:08:24.789168 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.865588 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:24.867193 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 92021us
I20260202 15:08:24.871026 12615 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:24.869623 14176 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:24.881845 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.881666 14176 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.882315 14193 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:24.882920 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:24.883064 14193 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.883585 14176 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:24.885375 14229 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:24.885893 14229 raft_consensus.cc:697] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:24.849061 14243 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:24.862704 14206 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:24.937580 12763 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:24.865952 14291 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:24.935882 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:24.933327 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 54901.1us
I20260202 15:08:24.938247 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:24.841220 14183 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:24.941254 12614 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:24.935171 12800 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
W20260202 15:08:24.945705 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:24.879076 14276 raft_consensus.cc:2804] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:24.921270 14155 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:24.887003 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:24.921273 14158 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.879730 14304 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:24.934970 14268 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:24.940933 14206 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:24.937206 14216 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:24.871011 14282 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.014274 14292 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:24.871552 14202 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:24.941354 14301 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:24.941386 14311 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:24.952409 14211 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:24.886711 14229 consensus_queue.cc:237] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.013659 12648 raft_consensus.cc:1240] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:25.014727 14287 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:25.029433 14176 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:25.043229 14312 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260202 15:08:25.043160 14220 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:25.044217 14227 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:25.015396 14232 raft_consensus.cc:1027] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: attempt to promote peer 66b8c715b9e6479e9218b08bd3156380: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260202 15:08:25.014205 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:24.945640 14291 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:25.018626 14279 consensus_meta.cc:287] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.000s user 0.006s sys 0.006s
I20260202 15:08:25.020807 14233 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:25.018599 14276 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:25.020828 14155 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:25.006304 14252 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260202 15:08:25.041074 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:24.934067 14214 consensus_queue.cc:237] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.097298 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.028395 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:25.013729 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1227) took 1667 ms. Trace:
W20260202 15:08:25.051604 14260 consensus_meta.cc:287] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.573s user 0.011s sys 0.000s
I20260202 15:08:25.104168 14260 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:25.104619 14260 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.074537 14320 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:25.105949 14320 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.106376 14260 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.043996 14193 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:25.040761 12792 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:25.081998 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:25.048630 14304 raft_consensus.cc:697] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:25.051153 14211 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:25.051805 14208 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:25.029660 14158 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:25.094209 14321 log.cc:927] Time spent T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260202 15:08:25.081614 14176 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.098241 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.098551 12763 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:25.075655 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.102845 12720 rpcz_store.cc:276] 0202 15:08:23.338516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.804661 (+1466145us) service_pool.cc:225] Handling call
0202 15:08:24.862318 (+ 57657us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:24.887244 (+ 24926us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:24.922830 (+ 35586us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:24.936558 (+ 13728us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:24.939404 (+ 2846us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:24.944977 (+ 5573us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:25.005395 (+ 60418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:25.110522 14265 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:25.089854 14276 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:25.064404 14268 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.109887 12798 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:25.092047 14155 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:25.091698 14292 raft_consensus.cc:697] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:25.135383 13328 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:25.078857 14287 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.136920 12763 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:25.130932 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:25.086012 14220 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.050943 14216 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.091576 14233 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.220480 13363 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260202 15:08:25.139245 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 194663us
I20260202 15:08:25.063014 14311 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.108770 14130 consensus_queue.cc:1059] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.059s
I20260202 15:08:25.269691 13326 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:25.114264 14193 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:25.168756 14254 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:25.271342 14326 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:25.271544 14254 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.271785 14326 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.168567 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:25.263887 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 128549us
I20260202 15:08:25.265305 12764 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:25.273805 14326 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.265800 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:25.274940 14254 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.275157 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:25.139705 14279 consensus_queue.cc:1059] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.142s
I20260202 15:08:25.275954 13363 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:25.227352 13362 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:25.116678 14260 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:25.267863 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 131045us
W20260202 15:08:25.260862 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:25.273995 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.243968 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:25.277508 12795 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 5
I20260202 15:08:25.277549 13328 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:25.087396 14227 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.286675 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.286582 14292 consensus_queue.cc:237] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.288160 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.289548 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:25.290160 12642 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:25.261689 14310 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:25.244144 14255 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:25.281512 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:25.235754 14322 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:25.118525 14211 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:25.276773 14260 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.220194 14209 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:25.278210 14243 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:25.292537 14326 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:25.170039 14113 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:25.225394 14319 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:25.310760 14318 log.cc:927] Time spent T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.281s user 0.000s sys 0.001s
I20260202 15:08:25.284471 14265 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:25.131999 14208 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.116247 14213 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:25.320964 14327 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:25.321099 12795 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:25.300377 14176 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.300606 14316 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:25.313872 14225 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:25.337260 12642 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:25.317451 14151 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:25.315806 14243 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:25.284806 14193 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:25.331034 14202 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.339860 14291 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:25.295071 14304 consensus_queue.cc:237] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.331297 14282 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:25.317569 14209 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:25.436239 14317 log.cc:927] Time spent T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.335s user 0.000s sys 0.001s
I20260202 15:08:25.414027 14243 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:25.322319 14310 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.326974 14319 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.323110 14326 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.331846 14265 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:25.456676 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 187039us
W20260202 15:08:25.458261 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 193019us
W20260202 15:08:25.513108 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 494026us
I20260202 15:08:25.438102 14202 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:25.513590 12691 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.022s sys 0.055s
W20260202 15:08:25.514207 12691 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s user 0.022s sys 0.055s
I20260202 15:08:25.442440 14282 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:25.521855 13326 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:25.517619 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:25.515475 13363 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 1 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:25.555508 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:25.342476 14255 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.516090 12795 raft_consensus.cc:1240] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:25.416508 14316 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.444489 14268 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.557664 13363 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
W20260202 15:08:25.558915 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 44485.3us
I20260202 15:08:25.500792 14215 raft_consensus.cc:1027] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: attempt to promote peer 66b8c715b9e6479e9218b08bd3156380: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260202 15:08:25.050344 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:25.554626 14301 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:25.563596 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1322) took 1375 ms. Trace:
I20260202 15:08:25.563665 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1321) took 1852 ms. Trace:
I20260202 15:08:25.602906 12799 rpcz_store.cc:276] 0202 15:08:23.711487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.929009 (+217522us) service_pool.cc:225] Handling call
0202 15:08:23.929356 (+ 347us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:23.929790 (+ 434us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:23.929828 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:23.941389 (+ 11561us) log.cc:844] Serialized 26 byte log entry
0202 15:08:24.857354 (+915965us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b1800363580. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471 00007f24ed51a73f
0202 15:08:25.117651 (+260297us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:25.513586 (+395935us) log.cc:844] Serialized 14 byte log entry
0202 15:08:25.550792 (+ 37206us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:25.550853 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:25.550888 (+ 35us) raft_consensus.cc:1685] finished
0202 15:08:25.563532 (+ 12644us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:25.563621 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49995,"raft.queue_time_us":42890,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":16619,"spinlock_wait_cycles":9072512,"thread_start_us":634110,"threads_started":2,"wal-append.queue_time_us":1277102}
I20260202 15:08:25.602851 12646 rpcz_store.cc:276] 0202 15:08:24.188442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.203201 (+ 14759us) service_pool.cc:225] Handling call
0202 15:08:24.219894 (+ 16693us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:24.288254 (+ 68360us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:24.294848 (+ 6594us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:24.297094 (+ 2246us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:24.302137 (+ 5043us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:25.563413 (+1261276us) spinlock_profiling.cc:243] Waited 57 ms on lock 0x7b54002a9c54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:25.563454 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:25.563554 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":131045248}
I20260202 15:08:25.606846 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.608003 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.608620 14173 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.608960 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.609479 12646 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:25.610222 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.610956 14301 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.612223 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.612805 14301 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:25.614645 14319 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.615796 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34778 (request call id 1413) took 2151 ms. Trace:
I20260202 15:08:25.564944 14316 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.626353 12615 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:25.616151 12650 rpcz_store.cc:276] 0202 15:08:23.464380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.464549 (+ 169us) service_pool.cc:225] Handling call
0202 15:08:25.615199 (+2150650us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54001decd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c90c0 00007f24ed567bba 00007f24f251858e 00007f24ebab7437 00007f24ebab73c2 00007f24ebab72f1 00007f24ebab726c 00007f24ebab6522 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:25.615704 (+ 505us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2973528960}
I20260202 15:08:25.565163 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:25.566208 14288 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:25.566924 14287 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:25.568526 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:25.572930 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1328) took 1321 ms. Trace:
I20260202 15:08:25.575872 14332 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:25.576454 14090 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:25.555716 14202 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.638866 14246 consensus_queue.cc:1059] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:25.639050 14349 raft_consensus.cc:2749] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:25.639099 14090 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.639827 14199 raft_consensus.cc:3055] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:25.640175 14199 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:25.640329 14202 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.501466 14216 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.320297 14322 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.501766 14158 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:25.550354 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1550) took 1377 ms. Trace:
W20260202 15:08:25.625273 12689 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.011s sys 0.005s
W20260202 15:08:25.652940 12689 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s user 0.011s sys 0.005s
W20260202 15:08:25.636699 14328 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260202 15:08:25.413447 14327 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.554915 14315 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:25.556588 14233 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:25.586299 14320 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.635270 12644 rpcz_store.cc:276] 0202 15:08:24.251150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.251270 (+ 120us) service_pool.cc:225] Handling call
0202 15:08:25.572201 (+1320931us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:25.572417 (+ 216us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:25.572599 (+ 182us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:25.572632 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:25.572689 (+ 57us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:25.572724 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:25.572783 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:25.572878 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63241088}
I20260202 15:08:25.637617 12800 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:25.637995 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 214) took 1253 ms. Trace:
W20260202 15:08:25.638511 14179 consensus_meta.cc:287] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.057s user 0.003s sys 0.007s
I20260202 15:08:25.646215 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:25.624383 14220 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:25.651440 12690 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.006s sys 0.017s
I20260202 15:08:25.557031 14282 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:25.656636 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:25.661634 12725 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:25.663126 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:25.641312 14168 raft_consensus.cc:2804] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:25.640864 14090 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:25.639701 14246 raft_consensus.cc:2955] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:25.638880 14332 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:25.640874 14199 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:25.669739 12650 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260202 15:08:26.083179 12688 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.010s sys 0.029s
W20260202 15:08:25.906826 13360 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.887s user 0.031s sys 0.027s
I20260202 15:08:25.668537 12641 rpcz_store.cc:276] 0202 15:08:24.384342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.388309 (+ 3967us) service_pool.cc:225] Handling call
0202 15:08:24.388535 (+ 226us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:24.406119 (+ 17584us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:25.637775 (+1231656us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b540008a2d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:25.637923 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39295872}
W20260202 15:08:25.715118 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 437589us
I20260202 15:08:26.266072 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:26.266995 13362 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:26.268121 12795 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
W20260202 15:08:26.269049 12690 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.242s user 0.025s sys 0.027s
W20260202 15:08:26.269744 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:25.756134 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:26.249397 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:26.104040 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:26.271667 12718 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
W20260202 15:08:25.870309 12724 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.004s sys 0.020s
I20260202 15:08:26.219132 14265 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:26.272933 12724 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.139s user 0.004s sys 0.042s
W20260202 15:08:25.649626 14154 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.030s sys 0.080s
I20260202 15:08:26.248962 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:26.273815 12793 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.007s sys 0.018s
W20260202 15:08:25.756371 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 65474.4us
I20260202 15:08:26.274560 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:26.275631 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:26.273257 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:25.764669 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:25.650408 14326 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:25.649741 14227 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:25.649991 14260 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:26.249629 12798 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.018s sys 0.006s
I20260202 15:08:26.443233 12764 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
W20260202 15:08:26.641500 12798 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.079s user 0.019s sys 0.006s
W20260202 15:08:26.641852 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 198730us
W20260202 15:08:26.266227 14205 consensus_meta.cc:287] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.581s user 0.011s sys 0.000s
I20260202 15:08:26.642431 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1655) took 1139 ms. Trace:
I20260202 15:08:26.525249 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1222) took 3402 ms. Trace:
W20260202 15:08:26.596745 14359 log.cc:927] Time spent T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20260202 15:08:26.643057 12615 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:25.799213 14173 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.481173 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.526135 14250 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.643071 12791 rpcz_store.cc:276] 0202 15:08:25.502955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.506781 (+ 3826us) service_pool.cc:225] Handling call
0202 15:08:25.580928 (+ 74147us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:25.581282 (+ 354us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:25.581318 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:25.581882 (+ 564us) log.cc:844] Serialized 26 byte log entry
0202 15:08:26.265013 (+683131us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:26.280011 (+ 14998us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.373908 (+ 93897us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:26.642240 (+268332us) raft_consensus.cc:1685] finished
0202 15:08:26.642262 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.642380 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1913,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":237779,"threads_started":1,"wal-append.queue_time_us":936662}
W20260202 15:08:26.397233 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 127457us
I20260202 15:08:26.644735 12720 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 1.
I20260202 15:08:26.525610 14322 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:26.645565 14288 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:26.530601 14276 consensus_meta.cc:287] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.252s user 0.000s sys 0.005s
I20260202 15:08:26.525038 14327 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:26.646102 14288 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:26.532205 14243 consensus_meta.cc:287] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.032s user 0.009s sys 0.000s
I20260202 15:08:26.646605 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.647315 14243 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:25.681775 14316 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.967s
I20260202 15:08:26.648337 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.648866 12725 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:26.649343 14243 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.649549 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.519016 14259 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:26.369918 13361 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:26.650187 14259 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:26.650470 14171 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:26.248759 12688 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.777s user 0.014s sys 0.030s
W20260202 15:08:26.156072 14284 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.998s user 0.012s sys 0.026s
I20260202 15:08:26.451321 14179 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:26.651080 14171 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:26.651685 14284 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.497s user 0.016s sys 0.053s
I20260202 15:08:25.653059 14311 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:25.801136 14330 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:26.434967 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:26.274369 12793 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.007s sys 0.018s
I20260202 15:08:25.810603 12763 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:26.111649 14151 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:26.655141 14248 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:26.429209 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.655956 12717 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:26.657027 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.657505 12717 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:26.406510 14168 raft_consensus.cc:697] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:26.659519 14168 consensus_queue.cc:237] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.661969 14190 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:26.369841 14341 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:26.662315 14190 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:25.664568 14187 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.000s sys 0.017s
W20260202 15:08:26.238509 14335 log.cc:927] Time spent T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
W20260202 15:08:26.498463 14211 consensus_meta.cc:287] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.181s user 0.005s sys 0.004s
W20260202 15:08:26.465279 12722 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.025s sys 0.049s
I20260202 15:08:26.523627 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:26.663482 12546 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45928 (request call id 316) took 3624 ms. Trace:
W20260202 15:08:26.276906 13360 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.257s user 0.035s sys 0.027s
W20260202 15:08:26.273428 14154 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.260s user 0.037s sys 0.087s
I20260202 15:08:25.656387 14192 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:26.107620 14309 raft_consensus.cc:993] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower bbf140958c3b4f339c512a9dc4be2dab to start an election
W20260202 15:08:26.237802 14155 consensus_meta.cc:287] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.957s user 0.003s sys 0.004s
I20260202 15:08:25.651065 14202 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:26.099053 12792 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.009s sys 0.023s
W20260202 15:08:26.393451 12691 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.875s user 0.011s sys 0.019s
I20260202 15:08:26.482547 14183 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.239686 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:25.846722 14193 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.385974 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:25.847427 14348 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.489287 12545 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45920 (request call id 307) took 3565 ms. Trace:
I20260202 15:08:26.175429 14255 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:26.664245 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 853668us
I20260202 15:08:26.110801 14349 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.277240 12718 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:26.644266 14173 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.651996 14179 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.256187 14301 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:26.680589 12726 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:26.094373 12719 raft_consensus.cc:1240] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
W20260202 15:08:26.180521 14334 log.cc:927] Time spent T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260202 15:08:26.646185 14276 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.662750 14190 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:26.653033 14330 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:26.650331 12725 raft_consensus.cc:2410] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260202 15:08:25.903257 14232 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.025s sys 0.000s
W20260202 15:08:26.392920 14304 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.014s sys 0.034s
I20260202 15:08:25.756405 14130 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:25.855623 14209 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:26.684947 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 442) took 1457 ms. Trace:
I20260202 15:08:26.274883 14282 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:26.489718 14206 consensus_meta.cc:287] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.444s user 0.008s sys 0.000s
I20260202 15:08:25.801513 14268 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
W20260202 15:08:26.602846 12614 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.007s sys 0.008s
I20260202 15:08:26.528581 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 216) took 2143 ms. Trace:
W20260202 15:08:26.779897 12614 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.022s user 0.008s sys 0.009s
W20260202 15:08:26.780296 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 1.07107e+06us
I20260202 15:08:26.285712 14265 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.645749 12690 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:26.237550 14229 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:26.107870 14347 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:26.767251 14173 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:26.250885 14293 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:26.767488 14260 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:26.787286 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 141618us
I20260202 15:08:26.680051 14350 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.123s
I20260202 15:08:26.787985 12690 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:26.669395 14151 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.664644 12688 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:26.677744 13363 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:26.791127 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 637) took 1120 ms. Trace:
I20260202 15:08:26.791445 13363 rpcz_store.cc:276] 0202 15:08:25.671121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.523404 (+852283us) service_pool.cc:225] Handling call
0202 15:08:26.791076 (+267672us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:26.775560 14179 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:25.866251 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:26.183145 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:26.797869 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 133286us
I20260202 15:08:26.156371 14158 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.527199 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1656) took 1024 ms. Trace:
W20260202 15:08:26.681066 12691 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.163s user 0.016s sys 0.021s
I20260202 15:08:26.782387 14206 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.682556 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.805141 12723 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:26.772290 14130 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:26.683462 12763 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:26.698932 14301 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:26.795828 14347 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:26.809636 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 126249us
I20260202 15:08:26.810314 12763 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:26.108103 14342 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:26.793754 14173 raft_consensus.cc:2749] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:26.684475 13362 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:25.682524 14208 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:26.532971 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:26.651934 14259 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.695919 14331 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.696225 12764 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:26.683722 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1568) took 1017 ms. Trace:
I20260202 15:08:26.369781 14344 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:25.846596 14310 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:26.663348 14250 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:26.699501 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 409) took 3571 ms. Trace:
I20260202 15:08:26.647571 12720 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:26.690085 12615 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:26.827157 13361 rpcz_store.cc:276] 0202 15:08:23.124592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.124750 (+ 158us) service_pool.cc:225] Handling call
0202 15:08:23.138007 (+ 13257us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:26.650348 (+3512341us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.695340 (+ 44992us) spinlock_profiling.cc:243] Waited 3.23 s on lock 0x7b5400640354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:26.695534 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7431864576}
W20260202 15:08:26.829054 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 139034us
I20260202 15:08:26.829900 12615 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:26.691139 12547 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39704 (request call id 238) took 3311 ms. Trace:
W20260202 15:08:26.665673 14187 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.585s user 0.000s sys 0.035s
I20260202 15:08:26.361389 14319 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.414s
I20260202 15:08:26.679652 14198 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:26.835798 14198 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:26.836282 14198 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:26.412894 14333 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.285s
I20260202 15:08:26.818992 14316 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:26.666749 14150 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:26.658643 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.481556 14338 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.826525 14301 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:26.765594 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:26.682945 12545 rpcz_store.cc:276] 0202 15:08:22.924020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:22.924203 (+ 183us) service_pool.cc:225] Handling call
0202 15:08:26.489211 (+3565008us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:08:23.675041 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:08:23.701481 (+ 26440us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:08:23.701519 (+ 38us) write_op.cc:435] Acquired schema lock
0202 15:08:23.701541 (+ 22us) tablet.cc:660] Decoding operations
0202 15:08:23.744565 (+ 43024us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:08:23.775173 (+ 30608us) write_op.cc:641] Partition lock acquired for write op
0202 15:08:23.783089 (+ 7916us) tablet.cc:683] Acquiring locks for 1 operations
0202 15:08:23.887992 (+104903us) tablet.cc:699] Row locks acquired
0202 15:08:23.912748 (+ 24756us) write_op.cc:260] PREPARE: finished
0202 15:08:23.941639 (+ 28891us) write_op.cc:270] Start()
0202 15:08:23.949881 (+ 8242us) write_op.cc:276] Timestamp: P: 1770044903941590 usec, L: 0
0202 15:08:23.963004 (+ 13123us) op_driver.cc:348] REPLICATION: starting
0202 15:08:23.975208 (+ 12204us) log.cc:844] Serialized 541 byte log entry
0202 15:08:24.348879 (+373671us) op_driver.cc:464] REPLICATION: finished
0202 15:08:24.405651 (+ 56772us) write_op.cc:301] APPLY: starting
0202 15:08:24.405731 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
0202 15:08:24.405812 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
0202 15:08:24.405917 (+ 105us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:08:24.406117 (+ 200us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:08:24.406246 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0202 15:08:24.406287 (+ 41us) write_op.cc:312] APPLY: finished
0202 15:08:24.406429 (+ 142us) log.cc:844] Serialized 23 byte log entry
0202 15:08:24.406550 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:24.406620 (+ 70us) write_op.cc:454] Released schema lock
0202 15:08:24.406653 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":1170382,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":43467,"apply.run_cpu_time_us":1319,"apply.run_wall_time_us":12898,"num_ops":1,"prepare.queue_time_us":83799,"prepare.run_cpu_time_us":3032,"prepare.run_wall_time_us":303106,"replication_time_us":351467,"thread_start_us":382040,"threads_started":1}]]}
W20260202 15:08:26.685842 14279 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.410s user 0.011s sys 0.021s
I20260202 15:08:26.273622 14258 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:26.681602 13357 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:25.765336 14315 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:26.759763 14232 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.661s user 0.039s sys 0.002s
I20260202 15:08:26.679894 14171 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.666409 12724 rpcz_store.cc:276] 0202 15:08:23.122340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.122629 (+ 289us) service_pool.cc:225] Handling call
0202 15:08:23.122991 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:26.273422 (+3150431us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:26.273454 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:26.273491 (+ 37us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:26.273517 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.524928 (+251411us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54000181d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:26.524986 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.525143 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":747,"raft.queue_time_us":443067,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":91,"spinlock_wait_cycles":4622407936,"thread_start_us":1139321,"threads_started":1}
I20260202 15:08:26.851238 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.838795 14320 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260202 15:08:26.697958 13355 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.896s user 0.007s sys 0.014s
I20260202 15:08:26.777640 14332 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.740154 14329 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.667814 14243 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:26.680823 14202 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.781140 12614 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:26.690531 14211 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.833951 14250 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.779966 12648 rpcz_store.cc:276] 0202 15:08:24.384845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.385077 (+ 232us) service_pool.cc:225] Handling call
0202 15:08:24.385414 (+ 337us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:25.063378 (+677964us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.528339 (+1464961us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b540008a7d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:26.528511 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1444482048}
I20260202 15:08:26.674269 14326 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:26.614490 14214 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.060s user 0.012s sys 0.019s
W20260202 15:08:26.867411 14214 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.329s user 0.012s sys 0.021s
I20260202 15:08:26.867367 14326 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:26.677788 14205 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:26.869597 14205 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:26.865485 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 84434.3us
W20260202 15:08:26.677302 12722 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.900s user 0.028s sys 0.049s
I20260202 15:08:26.772746 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1552) took 1109 ms. Trace:
I20260202 15:08:26.853559 14320 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:26.792770 12726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1243) took 2528 ms (client timeout 1912 ms). Trace:
I20260202 15:08:26.677515 12546 rpcz_store.cc:276] 0202 15:08:23.038762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.038903 (+ 141us) service_pool.cc:225] Handling call
0202 15:08:26.663430 (+3624527us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:08:23.421812 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:08:23.491544 (+ 69732us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:08:23.491576 (+ 32us) write_op.cc:435] Acquired schema lock
0202 15:08:23.491591 (+ 15us) tablet.cc:660] Decoding operations
0202 15:08:23.491876 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:08:23.491926 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0202 15:08:23.491949 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0202 15:08:23.492097 (+ 148us) tablet.cc:699] Row locks acquired
0202 15:08:23.492111 (+ 14us) write_op.cc:260] PREPARE: finished
0202 15:08:23.492225 (+ 114us) write_op.cc:270] Start()
0202 15:08:23.492316 (+ 91us) write_op.cc:276] Timestamp: P: 1770044903492194 usec, L: 0
0202 15:08:23.492333 (+ 17us) op_driver.cc:348] REPLICATION: starting
0202 15:08:23.492993 (+ 660us) log.cc:844] Serialized 541 byte log entry
0202 15:08:24.124501 (+631508us) op_driver.cc:464] REPLICATION: finished
0202 15:08:24.308760 (+184259us) write_op.cc:301] APPLY: starting
0202 15:08:24.313607 (+ 4847us) tablet.cc:1365] starting BulkCheckPresence
0202 15:08:24.325671 (+ 12064us) tablet.cc:1368] finished BulkCheckPresence
0202 15:08:24.331516 (+ 5845us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:08:24.377467 (+ 45951us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:08:24.398494 (+ 21027us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0202 15:08:24.401965 (+ 3471us) write_op.cc:312] APPLY: finished
0202 15:08:24.404677 (+ 2712us) log.cc:844] Serialized 23 byte log entry
0202 15:08:24.404843 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:24.404934 (+ 91us) write_op.cc:454] Released schema lock
0202 15:08:24.404972 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0202 15:08:24.405324 (+ 352us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:24.405350 (+ 26us) write_op.cc:454] Released schema lock
0202 15:08:24.759062 (+353712us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:24.759124 (+ 62us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":904167,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":173589,"apply.run_cpu_time_us":2015,"apply.run_wall_time_us":106043,"num_ops":1,"prepare.queue_time_us":148680,"prepare.run_cpu_time_us":27743,"prepare.run_wall_time_us":242644,"raft.queue_time_us":648090,"raft.run_cpu_time_us":74759,"raft.run_wall_time_us":651007,"replication_time_us":622955,"thread_start_us":741468,"threads_started":4,"wal-append.queue_time_us":183030,"wal-append.run_cpu_time_us":53344,"wal-append.run_wall_time_us":1728132}]]}
I20260202 15:08:26.796911 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:26.799371 12688 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:26.872575 12726 rpcz_store.cc:269] 0202 15:08:24.255817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.255939 (+ 122us) service_pool.cc:225] Handling call
0202 15:08:26.784268 (+2528329us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b54003282d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:26.784361 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5574447488}
W20260202 15:08:26.873106 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 73801.2us
I20260202 15:08:26.805807 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1564) took 1141 ms. Trace:
I20260202 15:08:26.878614 12723 rpcz_store.cc:276] 0202 15:08:25.664857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.682354 (+1017497us) service_pool.cc:225] Handling call
0202 15:08:26.805750 (+123396us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:26.682996 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:26.879737 12799 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:26.880254 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1658) took 1376 ms. Trace:
I20260202 15:08:26.880568 12799 rpcz_store.cc:276] 0202 15:08:25.503697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.604899 (+101202us) service_pool.cc:225] Handling call
0202 15:08:26.682883 (+1077984us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:26.880193 (+197310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59497216}
I20260202 15:08:26.784735 14268 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.804589 12797 rpcz_store.cc:276] 0202 15:08:25.503193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.506626 (+ 3433us) service_pool.cc:225] Handling call
0202 15:08:26.526294 (+1019668us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:26.526535 (+ 241us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:26.526786 (+ 251us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:26.526830 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:26.526891 (+ 61us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:26.526931 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.527007 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.527132 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65421696}
I20260202 15:08:26.805600 14193 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.683363 14349 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.795675 14209 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:26.820026 12763 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:26.773726 13358 rpcz_store.cc:276] 0202 15:08:25.227643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.281775 (+ 54132us) service_pool.cc:225] Handling call
0202 15:08:25.326682 (+ 44907us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:25.650943 (+324261us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:25.653896 (+ 2953us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:25.656144 (+ 2248us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:25.681564 (+ 25420us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.684729 (+1003165us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b540068ef54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:26.684778 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.684893 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":714683264}
I20260202 15:08:26.823033 12718 rpcz_store.cc:276] 0202 15:08:25.666192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.272989 (+606797us) service_pool.cc:225] Handling call
0202 15:08:26.683672 (+410683us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:26.890674 12642 raft_consensus.cc:1240] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:26.805737 14102 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:26.786403 14265 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:26.825156 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 129002us
I20260202 15:08:26.872265 14232 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:26.825721 14208 raft_consensus.cc:2804] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:25.652762 12721 rpcz_store.cc:276] 0202 15:08:24.128847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.134604 (+ 5757us) service_pool.cc:225] Handling call
0202 15:08:24.183212 (+ 48608us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:24.225824 (+ 42612us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:24.229964 (+ 4140us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:24.297748 (+ 67784us) log.cc:844] Serialized 26 byte log entry
0202 15:08:24.768018 (+470270us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:24.768057 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:24.768119 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:25.441511 (+673392us) raft_consensus.cc:1685] finished
0202 15:08:25.441539 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:25.506495 (+ 64956us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1961,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":10526,"thread_start_us":403132,"threads_started":1,"wal-append.queue_time_us":556110,"wal-append.run_cpu_time_us":2716,"wal-append.run_wall_time_us":1636560}
I20260202 15:08:26.916211 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.916765 12721 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:26.917762 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.918160 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 440) took 1968 ms. Trace:
W20260202 15:08:26.918432 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 216524us
I20260202 15:08:26.918802 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.830596 12547 rpcz_store.cc:276] 0202 15:08:23.379558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.379687 (+ 129us) service_pool.cc:225] Handling call
0202 15:08:26.691085 (+3311398us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":896335,"reported_tablets":1}
I20260202 15:08:26.786571 14229 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.919579 14171 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:26.826333 14331 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:26.923157 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:26.810146 14327 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.834985 12615 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:26.835843 14340 log.cc:927] Time spent T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.404s user 0.001s sys 0.001s
I20260202 15:08:26.680559 14309 raft_consensus.cc:1081] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Signalling peer bbf140958c3b4f339c512a9dc4be2dab to start an election
I20260202 15:08:26.806593 13359 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:26.794771 14260 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.839725 14366 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:26.820525 14227 raft_consensus.cc:2749] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:26.824218 14179 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:26.680955 12792 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.097s user 0.018s sys 0.046s
I20260202 15:08:26.843344 13328 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:26.674840 14341 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:08:26.845227 14279 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.569s user 0.016s sys 0.021s
I20260202 15:08:26.846961 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:26.841392 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1560) took 1175 ms. Trace:
I20260202 15:08:26.847075 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1652) took 1345 ms. Trace:
I20260202 15:08:26.808300 14158 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.851734 12724 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:26.961853 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:26.765574 14304 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.110s user 0.018s sys 0.038s
I20260202 15:08:26.841248 14154 raft_consensus.cc:2749] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:26.848994 14258 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.819386 14293 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:26.850281 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 219) took 2101 ms (client timeout 2220 ms). Trace:
I20260202 15:08:26.966383 12724 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:26.965970 12800 raft_consensus.cc:2955] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:26.831377 14344 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:26.831533 14342 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.861653 14130 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.863054 14151 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:26.854565 13355 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.054s user 0.008s sys 0.014s
I20260202 15:08:26.681200 14155 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.855273 14206 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:26.850576 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 635) took 1183 ms. Trace:
I20260202 15:08:26.863494 14329 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.869114 14326 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.680459 14192 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.867538 14282 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.883610 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.984174 12723 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:26.863929 14243 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.885473 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.848035 14316 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.852763 14330 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:26.900772 14354 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:26.991354 14354 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:26.991824 14354 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:26.992091 14267 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260202 15:08:26.901592 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 81618us
I20260202 15:08:26.682601 14348 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.907584 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:26.910871 14292 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.621s user 0.009s sys 0.052s
I20260202 15:08:26.977232 14370 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:26.882861 14320 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:26.921654 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:26.921970 13326 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:26.922297 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:26.922497 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1654) took 1419 ms. Trace:
I20260202 15:08:26.922933 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:27.006057 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 84135.5us
I20260202 15:08:26.924243 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:26.902894 14287 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.918643 13360 rpcz_store.cc:276] 0202 15:08:24.949540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.949907 (+ 367us) service_pool.cc:225] Handling call
0202 15:08:24.954081 (+ 4174us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:24.954284 (+ 203us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:26.677661 (+1723377us) log.cc:844] Serialized 14 byte log entry
0202 15:08:26.917666 (+240005us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:26.917738 (+ 72us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:26.917803 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.917917 (+ 114us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.918058 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":328034,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":5945,"raft.queue_time_us":1545,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"thread_start_us":1897177,"threads_started":2}
W20260202 15:08:27.008056 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 173155us
I20260202 15:08:27.008620 12799 raft_consensus.cc:2393] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:26.847612 14338 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.011229 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:26.928503 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:26.928664 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:26.929126 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 447) took 1100 ms. Trace:
I20260202 15:08:26.929979 14306 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:26.925799 14058 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:26.930512 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:26.908303 14150 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:26.932034 14265 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:27.004123 14370 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:26.932435 14168 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:26.915653 14208 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:26.903541 14209 raft_consensus.cc:697] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:26.907543 14232 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:26.886094 14187 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260202 15:08:26.918681 14350 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:26.881702 12719 rpcz_store.cc:276] 0202 15:08:25.661484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.666410 (+ 4926us) service_pool.cc:225] Handling call
0202 15:08:25.866399 (+199989us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:26.722327 (+855928us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.770540 (+ 48213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:26.944646 14179 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.926224 14276 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.021370 14158 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.013020 14287 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:26.938006 14327 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.946446 13361 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:26.953702 14250 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:26.952514 14211 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:26.951578 14229 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:26.946141 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 636) took 1274 ms. Trace:
I20260202 15:08:26.945298 14198 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:26.960172 14341 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.960419 14205 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.963752 14303 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:26.962893 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 119638us
I20260202 15:08:27.054741 13328 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:27.054466 14303 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.055605 13328 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:27.056345 14303 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.057700 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.051630 13359 rpcz_store.cc:276] 0202 15:08:25.667273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.800924 (+133651us) service_pool.cc:225] Handling call
0202 15:08:26.939869 (+1138945us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:26.965818 14220 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:26.924659 14171 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.058825 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.059464 13359 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:27.060087 14171 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:26.968948 12643 rpcz_store.cc:269] 0202 15:08:24.746371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.748612 (+ 2241us) service_pool.cc:225] Handling call
0202 15:08:26.532798 (+1784186us) spinlock_profiling.cc:243] Waited 6.07 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:26.844996 (+312198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13963136}
I20260202 15:08:27.060240 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.060765 13359 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:27.061028 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.062165 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:26.983489 12641 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:26.959127 12720 rpcz_store.cc:276] 0202 15:08:25.663756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.646350 (+982594us) service_pool.cc:225] Handling call
0202 15:08:26.838679 (+192329us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:26.986716 12649 raft_consensus.cc:2393] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:26.961036 12793 rpcz_store.cc:276] 0202 15:08:25.502158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.506729 (+ 4571us) service_pool.cc:225] Handling call
0202 15:08:25.601536 (+ 94807us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:25.606641 (+ 5105us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:25.606678 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:25.607202 (+ 524us) log.cc:844] Serialized 26 byte log entry
0202 15:08:26.655136 (+1047934us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:26.655179 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.655253 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:26.838272 (+183019us) raft_consensus.cc:1685] finished
0202 15:08:26.840699 (+ 2427us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.847007 (+ 6308us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1600,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":110,"thread_start_us":1047548,"threads_started":1,"wal-append.queue_time_us":1181701}
I20260202 15:08:26.969697 14374 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:26.994554 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1561) took 1330 ms. Trace:
I20260202 15:08:26.996096 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1559) took 1332 ms. Trace:
I20260202 15:08:26.975478 14202 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.978197 13357 rpcz_store.cc:276] 0202 15:08:25.666684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.855749 (+189065us) service_pool.cc:225] Handling call
0202 15:08:26.849485 (+993736us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:27.066843 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.066964 12723 rpcz_store.cc:276] 0202 15:08:25.663505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.990152 (+1326647us) service_pool.cc:225] Handling call
0202 15:08:26.995505 (+ 5353us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:26.995710 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:26.995747 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:26.995800 (+ 53us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:26.995842 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.995921 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.996034 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:27.067534 12793 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:27.067654 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.068291 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.068234 14190 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.068964 12793 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:27.069754 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.069825 14199 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:27.070199 14190 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.070242 12793 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:27.071002 12795 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:27.066838 12726 rpcz_store.cc:276] 0202 15:08:25.664031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.993666 (+1329635us) service_pool.cc:225] Handling call
0202 15:08:26.993984 (+ 318us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:26.994157 (+ 173us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:26.994188 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:26.994259 (+ 71us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:26.994303 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.994373 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.994489 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:27.015576 13358 rpcz_store.cc:276] 0202 15:08:25.828794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.923648 (+1094854us) service_pool.cc:225] Handling call
0202 15:08:26.928666 (+ 5018us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:26.928829 (+ 163us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:26.928864 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:26.928900 (+ 36us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:26.928931 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.928991 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.929082 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:27.072453 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:27.072953 13358 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:26.911638 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 464183us
I20260202 15:08:26.809741 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1663) took 1166 ms. Trace:
I20260202 15:08:27.066593 14374 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.979979 14347 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:26.928253 12689 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:26.973932 14304 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260202 15:08:27.006865 13363 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:27.010313 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:27.003257 14292 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.711s user 0.010s sys 0.053s
I20260202 15:08:27.013624 12615 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:27.005882 12798 rpcz_store.cc:276] 0202 15:08:25.502689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.502817 (+ 128us) service_pool.cc:225] Handling call
0202 15:08:25.503053 (+ 236us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:25.503212 (+ 159us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:26.653643 (+1150431us) log.cc:844] Serialized 14 byte log entry
0202 15:08:26.922059 (+268416us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:26.922135 (+ 76us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:26.922222 (+ 87us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.922297 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:26.922426 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":533643,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":565,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":10975,"spinlock_wait_cycles":1144576,"thread_start_us":1347313,"threads_started":2,"wal-append.queue_time_us":291469}
I20260202 15:08:27.172465 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.173094 12798 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:27.012849 14254 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.029304 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:27.013758 14268 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.010618 14369 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:27.178826 14347 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:27.013909 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 424) took 3407 ms. Trace:
I20260202 15:08:27.179309 14347 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:26.936687 14368 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
W20260202 15:08:26.944600 14351 log.cc:927] Time spent T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.442s user 0.003s sys 0.000s
W20260202 15:08:27.038990 14362 log.cc:927] Time spent T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260202 15:08:27.028891 14168 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.028271 14370 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:27.028410 14150 raft_consensus.cc:697] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:27.048178 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1557) took 1385 ms. Trace:
I20260202 15:08:27.191886 14150 consensus_queue.cc:237] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:27.191979 12722 rpcz_store.cc:276] 0202 15:08:25.663030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.663121 (+ 91us) service_pool.cc:225] Handling call
0202 15:08:25.663321 (+ 200us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:25.777091 (+113770us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:26.872065 (+1094974us) log.cc:844] Serialized 14 byte log entry
0202 15:08:26.917378 (+ 45313us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:26.917424 (+ 46us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:26.917465 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.048014 (+130549us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b540009f1d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:27.048057 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.048131 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1060252,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":70,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"spinlock_wait_cycles":261118336,"thread_start_us":1139674,"threads_started":2,"wal-append.queue_time_us":141306}
I20260202 15:08:27.193943 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.030473 14208 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.167868 14344 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.029264 12717 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:27.018071 14306 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:27.199965 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 632) took 2082 ms. Trace:
I20260202 15:08:27.200063 14306 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.034121 14232 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.018249 14058 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:27.021592 14293 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.053735 14155 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.053644 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.039312 14350 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.047297 14209 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:27.058267 13361 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:27.039279 14192 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.178857 14185 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:27.056308 14198 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.062786 14243 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.003299 12763 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:27.039496 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 434) took 2658 ms. Trace:
I20260202 15:08:27.164491 12794 rpcz_store.cc:276] 0202 15:08:25.643560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.838084 (+194524us) service_pool.cc:225] Handling call
0202 15:08:26.809688 (+971604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:27.008507 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.167469 14282 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
W20260202 15:08:27.167606 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 239405us
I20260202 15:08:27.225950 14316 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.171342 12647 raft_consensus.cc:2368] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3.
I20260202 15:08:27.062729 12643 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:27.173857 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:26.901170 14193 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.271278 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.068856 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.234478 12791 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:27.061571 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:27.176759 12796 raft_consensus.cc:1240] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:27.273288 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.273955 12798 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:27.177314 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.175469 14342 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.178267 14348 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:27.075074 14329 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.165593 14276 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:27.276751 13359 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.277050 14276 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:27.277310 14058 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.277442 14276 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.184038 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.278412 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.278905 14288 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.279388 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.279946 14287 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.280292 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.281235 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.282163 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.171206 14158 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:27.168629 14171 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:27.285960 14171 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:27.286334 14171 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.287281 14276 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.288357 13357 raft_consensus.cc:2368] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4.
I20260202 15:08:27.288904 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34778 (request call id 1414) took 1652 ms. Trace:
W20260202 15:08:27.201146 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:27.201184 12644 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:27.289458 14158 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:27.289867 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:27.250680 14185 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.064067 14220 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:27.290392 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.260552 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1563) took 1592 ms. Trace:
I20260202 15:08:27.291067 12644 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:27.291404 14185 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.291512 14220 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.291839 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.200333 13354 rpcz_store.cc:276] 0202 15:08:25.117655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.119677 (+ 2022us) service_pool.cc:225] Handling call
0202 15:08:25.234802 (+115125us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:25.290584 (+ 55782us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:25.300298 (+ 9714us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:25.336660 (+ 36362us) log.cc:844] Serialized 26 byte log entry
0202 15:08:25.753020 (+416360us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:25.775195 (+ 22175us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:25.799138 (+ 23943us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:26.870659 (+1071521us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:27.199705 (+329046us) raft_consensus.cc:1685] finished
0202 15:08:27.199729 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.199910 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2904,"raft.queue_time_us":355464,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":5212,"spinlock_wait_cycles":91633920,"thread_start_us":232763,"threads_started":1,"wal-append.queue_time_us":1009624}
I20260202 15:08:27.292770 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.183457 14229 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:27.178617 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.296380 12614 leader_election.cc:400] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4.
I20260202 15:08:27.296777 12614 leader_election.cc:403] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260202 15:08:27.297116 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.297696 14187 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.298067 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:27.237941 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1343) took 2383 ms (client timeout 1653 ms). Trace:
I20260202 15:08:27.281956 14327 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:27.298832 12645 rpcz_store.cc:269] 0202 15:08:24.854134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.882660 (+ 28526us) service_pool.cc:225] Handling call
0202 15:08:27.237709 (+2355049us) spinlock_profiling.cc:243] Waited 2.09 s on lock 0x7b540024e854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:27.237861 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4797928704}
I20260202 15:08:27.299726 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.300545 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.300484 14320 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.301141 12645 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:27.301769 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.301888 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.302254 14320 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:27.238677 13362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 638) took 1390 ms (client timeout 1838 ms). Trace:
I20260202 15:08:27.302639 12794 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
W20260202 15:08:27.302893 13362 rpcz_store.cc:269] 0202 15:08:25.848555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.283867 (+435312us) service_pool.cc:225] Handling call
0202 15:08:27.238519 (+954652us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b54006ade54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:27.238620 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71856896}
I20260202 15:08:27.303870 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.304399 13362 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:27.305125 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.305635 13362 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:27.306308 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.290434 14179 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:27.306864 14193 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.291208 12719 rpcz_store.cc:276] 0202 15:08:25.664588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.048070 (+1383482us) service_pool.cc:225] Handling call
0202 15:08:27.176691 (+128621us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:27.188996 (+ 12305us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:27.190372 (+ 1376us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:27.226190 (+ 35818us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:27.229755 (+ 3565us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.235108 (+ 5353us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.257048 (+ 21940us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:27.240195 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.307930 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.308352 14229 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.308519 12719 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:27.308753 14229 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:27.239478 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1348) took 2156 ms. Trace:
I20260202 15:08:27.309798 12642 rpcz_store.cc:276] 0202 15:08:25.082991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.423745 (+340754us) service_pool.cc:225] Handling call
0202 15:08:25.424038 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:26.891065 (+1467027us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.239234 (+348169us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54001de7d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:27.239415 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3372546432}
I20260202 15:08:27.310741 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:27.249400 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:27.311020 13362 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:27.235448 14374 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.226581 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:27.257697 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 254478us
I20260202 15:08:27.262027 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:27.169390 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 103695us
I20260202 15:08:27.317183 12764 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:27.317921 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.256945 13360 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:27.267818 12689 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:27.268110 12765 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:27.231616 14202 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:27.183571 14254 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.271859 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.273320 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.274735 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:27.177181 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 163603us
I20260202 15:08:27.325033 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.275904 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.325966 12647 raft_consensus.cc:2393] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:27.326187 12791 raft_consensus.cc:2410] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:27.254585 14347 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.326644 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:27.305650 12794 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260202 15:08:27.188086 14368 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.063196 12650 raft_consensus.cc:2393] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:27.294000 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.292783 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.266848 14293 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:27.295735 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.308007 12722 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:27.189975 14168 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:27.311232 12642 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:27.302596 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.205442 14349 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:27.313153 13326 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:27.315927 12688 leader_election.cc:400] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3.
I20260202 15:08:27.343137 12688 leader_election.cc:403] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260202 15:08:27.309347 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.318958 12763 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:27.319545 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 51835.5us
I20260202 15:08:27.252696 13356 rpcz_store.cc:276] 0202 15:08:24.381414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.390397 (+ 8983us) service_pool.cc:225] Handling call
0202 15:08:26.655428 (+2265031us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:26.655603 (+ 175us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:26.655896 (+ 293us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:26.655922 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:26.656323 (+ 401us) log.cc:844] Serialized 26 byte log entry
0202 15:08:26.942258 (+285935us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:26.971685 (+ 29427us) log.cc:844] Serialized 14 byte log entry
0202 15:08:26.998809 (+ 27124us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.009997 (+ 11188us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:27.015254 (+ 5257us) raft_consensus.cc:1685] finished
0202 15:08:27.030852 (+ 15598us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.035279 (+ 4427us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13437,"prepare.queue_time_us":12292,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":157796,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":25107,"spinlock_wait_cycles":3040512,"thread_start_us":269764,"threads_started":1,"wal-append.queue_time_us":302966}
I20260202 15:08:27.348093 13326 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:27.320669 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 52650.2us
I20260202 15:08:27.321141 14202 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.339411 14368 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.352015 14202 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:27.353344 14368 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.355398 12688 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:27.356186 14248 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.358903 14259 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:27.359067 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.359411 14259 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.359861 14259 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.359822 12719 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:27.361001 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.361689 12719 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:27.363130 12688 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:27.321503 14390 log.cc:927] Time spent T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260202 15:08:27.322093 14190 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:27.381103 14193 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:27.381232 14190 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:27.381677 14193 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.381822 14190 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:27.382210 14193 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:27.383056 14349 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:27.383518 14349 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:27.383927 14349 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.321615 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.387627 13360 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:27.321791 14327 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:27.322865 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.404899 14327 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:27.405292 14327 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:27.405694 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.179641 13355 rpcz_store.cc:276] 0202 15:08:23.606242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:23.684824 (+ 78582us) service_pool.cc:225] Handling call
0202 15:08:23.688150 (+ 3326us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:23.774930 (+ 86780us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:23.777743 (+ 2813us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:25.549360 (+1771617us) spinlock_profiling.cc:243] Waited 55.6 ms on lock 0x7b2c0004caa8. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ebe5d9b1 00007f24ebe6cfe2 00007f24ed59a851 00007f24ed55f0ba 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22
0202 15:08:25.653652 (+104292us) log.cc:844] Serialized 26 byte log entry
0202 15:08:26.978870 (+1325218us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:26.983814 (+ 4944us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.989432 (+ 5618us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:26.999021 (+ 9589us) raft_consensus.cc:1685] finished
0202 15:08:26.999047 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.011967 (+ 12920us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60173,"raft.queue_time_us":37264,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":83136,"spinlock_wait_cycles":127858048,"thread_start_us":1174643,"threads_started":1,"wal-append.queue_time_us":1305162}
I20260202 15:08:27.417043 14176 raft_consensus.cc:3055] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:27.417543 14176 raft_consensus.cc:740] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:27.168951 14250 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.423135 14250 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.327567 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 215) took 2520 ms. Trace:
I20260202 15:08:27.327984 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.425377 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.289729 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.430821 14176 consensus_queue.cc:260] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.432027 14176 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.330312 12764 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
W20260202 15:08:27.433621 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 103389us
I20260202 15:08:27.436587 14185 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:27.437045 14185 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:27.437450 14185 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:27.438550 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1562) took 1774 ms. Trace:
I20260202 15:08:27.330538 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:27.438973 12718 rpcz_store.cc:276] 0202 15:08:25.664329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.997699 (+1333370us) service_pool.cc:225] Handling call
0202 15:08:27.012228 (+ 14529us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:27.014464 (+ 2236us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:27.014491 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:27.016053 (+ 1562us) log.cc:844] Serialized 564 byte log entry
0202 15:08:27.031367 (+ 15314us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:27.031398 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.031454 (+ 56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:27.438330 (+406876us) raft_consensus.cc:1685] finished
0202 15:08:27.438356 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.438479 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":15090,"threads_started":1,"wal-append.queue_time_us":155489}
I20260202 15:08:27.316967 13359 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:27.440049 13362 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.440347 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.440856 12718 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:27.443889 13355 raft_consensus.cc:1240] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:27.445439 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 450) took 1083 ms. Trace:
I20260202 15:08:27.332938 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.446578 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:27.447304 13356 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.447927 12724 raft_consensus.cc:1240] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260202 15:08:27.450160 12724 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:27.338099 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.452289 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.452915 12648 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:27.338617 12644 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:27.339610 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.454434 12722 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:27.363624 14293 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:27.342520 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:27.457011 12723 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.344393 12689 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:27.289865 14158 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.345764 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.335227 14343 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:27.350188 12765 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:27.289391 12646 rpcz_store.cc:276] 0202 15:08:25.636198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.640715 (+ 4517us) service_pool.cc:225] Handling call
0202 15:08:27.288206 (+1647491us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540024e354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c90c0 00007f24ed567bba 00007f24f251858e 00007f24ebab7437 00007f24ebab73c2 00007f24ebab72f1 00007f24ebab726c 00007f24ebab6522 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:27.288799 (+ 593us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2670600576}
I20260202 15:08:27.414721 12643 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:27.418325 12688 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:27.427260 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1328) took 2201 ms. Trace:
I20260202 15:08:27.378767 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.424048 12796 rpcz_store.cc:276] 0202 15:08:24.806874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:24.807934 (+ 1060us) service_pool.cc:225] Handling call
0202 15:08:24.849350 (+ 41416us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:27.273632 (+2424282us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.327365 (+ 53733us) spinlock_profiling.cc:243] Waited 2.33 s on lock 0x7b54002d3a54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:27.327490 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5352206592}
I20260202 15:08:27.428514 13354 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:27.428886 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:27.419203 14345 consensus_queue.cc:1059] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
W20260202 15:08:27.457877 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 113604us
I20260202 15:08:27.467751 12648 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260202 15:08:27.555958 12689 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:27.471324 12642 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:27.556941 12689 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:27.558283 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 119363us
I20260202 15:08:27.362651 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:27.476131 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 126037us
I20260202 15:08:27.445894 13355 rpcz_store.cc:276] 0202 15:08:26.362490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.415424 (+1052934us) service_pool.cc:225] Handling call
0202 15:08:27.443626 (+ 28202us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:27.444410 (+ 784us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.445200 (+ 790us) spinlock_profiling.cc:243] Waited 27.8 ms on lock 0x7b54005eddd0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:27.445367 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64021888}
I20260202 15:08:27.562172 12765 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:27.455849 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.491631 12614 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:27.558982 12690 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:27.561324 12689 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:27.565510 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:27.566236 12690 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:27.567045 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 73775.5us
I20260202 15:08:27.475405 12722 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260202 15:08:27.569458 13358 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:27.569934 14347 raft_consensus.cc:2764] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260202 15:08:27.571465 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 79904.3us
I20260202 15:08:27.572975 12614 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:27.574648 12689 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:27.576439 13326 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:27.577422 13326 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:27.476318 14268 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:27.475893 14343 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:27.588819 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.581961 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:27.582764 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.583501 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1558) took 1920 ms. Trace:
I20260202 15:08:27.549448 14292 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:27.588970 14309 raft_consensus.cc:2749] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:27.596496 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.597131 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.597724 13363 raft_consensus.cc:2393] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:08:27.597987 12649 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1342) took 2748 ms (client timeout 1929 ms). Trace:
I20260202 15:08:27.598124 14345 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:27.563881 12720 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:27.598604 14345 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.583387 14215 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:27.599720 14215 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:27.600045 14215 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.600147 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.588394 14368 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:27.600864 12793 raft_consensus.cc:2393] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:27.601317 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.601670 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.601852 12796 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 1.
I20260202 15:08:27.602412 12793 raft_consensus.cc:2410] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:27.602579 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:27.603085 12796 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.604113 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.604521 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1566) took 1939 ms. Trace:
I20260202 15:08:27.604887 12717 rpcz_store.cc:276] 0202 15:08:25.665570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.658439 (+992869us) service_pool.cc:225] Handling call
0202 15:08:27.604330 (+945891us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b5400250dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:27.604464 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":426222976}
I20260202 15:08:27.605253 14259 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:27.606953 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.548645 14276 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:27.608098 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.608891 12791 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:27.609001 14338 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.609575 14338 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:27.609776 14315 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:27.609889 14338 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:27.610285 14338 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.610489 14260 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.610982 14260 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:27.611255 14260 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:27.611542 14260 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:27.612159 14350 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.613190 14171 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.613965 14202 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:27.614486 14202 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.605297 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.614856 14202 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.614984 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:27.615175 12646 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:27.615998 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.616040 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.616606 12648 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:27.617033 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:27.617530 12646 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.617290 14232 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.618284 12645 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:27.618402 14331 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.618873 14331 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:27.619135 14331 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.619261 14306 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.619443 14331 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:27.616204 14199 consensus_meta.cc:287] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.546s user 0.003s sys 0.022s
I20260202 15:08:27.619786 14306 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:27.621364 14187 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4.
I20260202 15:08:27.621454 14373 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:27.599264 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:27.598348 12649 rpcz_store.cc:269] 0202 15:08:24.849867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.023108 (+173241us) service_pool.cc:225] Handling call
0202 15:08:27.597825 (+2574717us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b540007a8d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:27.597930 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3936462080}
I20260202 15:08:27.595144 12725 rpcz_store.cc:276] 0202 15:08:25.663283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.770643 (+1107360us) service_pool.cc:225] Handling call
0202 15:08:26.771183 (+ 540us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:26.824464 (+ 53281us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:26.824506 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:26.832864 (+ 8358us) log.cc:844] Serialized 26 byte log entry
0202 15:08:26.916517 (+ 83653us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:26.916559 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.572369 (+655810us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b5400159354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:27.572406 (+ 37us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:27.572443 (+ 37us) raft_consensus.cc:1685] finished
0202 15:08:27.572455 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.583446 (+ 10991us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2966,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":8568,"spinlock_wait_cycles":121481600,"thread_start_us":62388,"threads_started":1,"wal-append.queue_time_us":328453}
I20260202 15:08:27.625082 13363 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260202 15:08:27.619362 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.597556 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.625978 12645 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
from {username='slave'} at 127.0.0.1:34784
I20260202 15:08:27.628504 12645 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:27.628806 12645 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.629379 12796 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:27.564818 14287 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:27.580427 14209 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:27.641719 14229 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:27.642843 12642 raft_consensus.cc:1240] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:27.644035 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.646188 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.646409 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.600893 14368 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:27.584718 14268 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:27.588784 14227 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:27.622143 14343 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:27.654196 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:27.647997 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 51763.4us
I20260202 15:08:27.621364 14306 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.695819 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.696683 12795 raft_consensus.cc:2410] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:27.588580 14320 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:27.600220 14345 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.623011 14292 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:27.693790 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 120888us
I20260202 15:08:27.696015 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:27.638046 14287 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:27.675578 14368 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:27.647924 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:27.753489 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 225) took 2009 ms (client timeout 1981 ms). Trace:
W20260202 15:08:27.753931 12641 rpcz_store.cc:269] 0202 15:08:25.708638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.265818 (+557180us) service_pool.cc:225] Handling call
0202 15:08:27.718264 (+1452446us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b5400299d54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:27.718361 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1648196864}
I20260202 15:08:27.711771 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 639) took 1439 ms. Trace:
I20260202 15:08:27.756238 13360 rpcz_store.cc:276] 0202 15:08:26.272087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.711131 (+1439044us) service_pool.cc:225] Handling call
0202 15:08:27.711388 (+ 257us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:27.711533 (+ 145us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:27.711552 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:27.711588 (+ 36us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:27.711610 (+ 22us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.711651 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.711728 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:27.676132 14254 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.764842 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.765516 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.772199 12642 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:27.766094 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.772806 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:27.770325 13328 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:27.773293 13360 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:27.773981 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:27.774439 13360 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.779363 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:27.779687 14208 consensus_meta.cc:287] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.585s user 0.017s sys 0.012s
I20260202 15:08:27.780355 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.781212 12614 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:27.781834 14329 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:27.781967 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.782186 14329 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:27.782528 14329 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.783056 14208 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.784408 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.784718 12797 raft_consensus.cc:1240] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:27.780299 14158 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.693667 14232 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.793293 13328 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:27.796526 13328 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:27.798107 13328 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:27.801385 13328 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:27.737236 14320 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:27.676353 14268 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.753229 14292 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.754956 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.712627 14363 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:27.767112 14287 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.809597 13328 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:27.825721 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 251290us
I20260202 15:08:27.826501 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.826463 12689 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:27.689162 14215 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.647563 14373 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:27.790516 14259 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.808260 14350 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:27.703930 14227 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.808636 14347 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:27.818760 14370 consensus_meta.cc:287] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.624s user 0.009s sys 0.045s
I20260202 15:08:27.790735 13355 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:27.818817 14320 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:27.814229 14374 raft_consensus.cc:2804] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:27.816061 14276 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:27.847321 12689 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:27.848626 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1665) took 1571 ms. Trace:
I20260202 15:08:27.854741 12800 rpcz_store.cc:276] 0202 15:08:26.277438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.277553 (+ 115us) service_pool.cc:225] Handling call
0202 15:08:26.277818 (+ 265us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:26.965864 (+688046us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:08:26.985676 (+ 19812us) log.cc:844] Serialized 14 byte log entry
0202 15:08:27.006448 (+ 20772us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:27.006491 (+ 43us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:27.006526 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.715908 (+709382us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b1800472c60. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea47f2d6 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49
0202 15:08:27.848398 (+132490us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54003570d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:27.848451 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.848569 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":179,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":7529,"spinlock_wait_cycles":1585593984,"thread_start_us":18243,"threads_started":1,"wal-append.queue_time_us":63407}
I20260202 15:08:27.824837 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:27.865685 12643 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:27.866112 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:27.866814 13358 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:27.867843 13358 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260202 15:08:27.871280 14276 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:27.871775 14276 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:27.874260 14101 raft_consensus.cc:3055] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:27.874648 14101 raft_consensus.cc:740] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:27.875401 14101 consensus_queue.cc:260] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.876432 14101 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:27.877465 14192 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:27.834947 14363 raft_consensus.cc:697] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:27.882380 14363 consensus_queue.cc:237] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:27.883723 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 448) took 1977 ms. Trace:
I20260202 15:08:27.884974 13354 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:27.888159 12796 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:27.860617 14215 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:27.861925 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:27.862860 12649 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:27.864037 14208 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:27.864986 14393 log.cc:927] Time spent T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260202 15:08:27.943753 12649 raft_consensus.cc:606] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:27.846743 14250 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:27.849498 14179 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:27.851477 14319 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:27.889866 14354 consensus_meta.cc:287] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.898s user 0.007s sys 0.013s
I20260202 15:08:27.889274 14350 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:27.946570 14259 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:27.942819 14374 raft_consensus.cc:697] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:27.954013 14215 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:27.850644 14370 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:27.995184 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 414040us
I20260202 15:08:27.884061 13361 rpcz_store.cc:276] 0202 15:08:25.906783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.324865 (+1418082us) service_pool.cc:225] Handling call
0202 15:08:27.882830 (+557965us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:27.883000 (+ 170us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:27.883025 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:27.883061 (+ 36us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:27.883086 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.883143 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.883585 (+ 442us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b54005131d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:27.883673 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1280320384}
I20260202 15:08:28.003644 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:28.005636 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:27.999903 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 123644us
I20260202 15:08:28.005089 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:27.999914 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:28.010037 12689 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:27.999701 13362 consensus_meta.cc:287] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.559s user 0.016s sys 0.010s
W20260202 15:08:28.003037 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 436817us
I20260202 15:08:28.016358 13362 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:28.016804 12690 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:28.019155 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 600867us
I20260202 15:08:28.024638 12614 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:27.988708 14259 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:28.032727 12688 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.027850 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:28.033910 12726 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:28.034991 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:28.036643 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:28.037001 12615 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:28.040326 13363 raft_consensus.cc:1240] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:27.997195 14215 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:27.960738 14319 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:27.987335 14354 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.048887 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 438) took 1033 ms. Trace:
I20260202 15:08:28.052089 13363 rpcz_store.cc:276] 0202 15:08:27.014991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.635946 (+620955us) service_pool.cc:225] Handling call
0202 15:08:27.636178 (+ 232us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:28.040935 (+404757us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:28.048675 (+ 7740us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b540041bed4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:28.048820 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":924576128}
I20260202 15:08:28.053503 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.055485 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.059566 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:28.034986 14229 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.036322 14259 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.032910 14171 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.062809 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:28.045065 14058 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:28.042502 14158 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.067904 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:28.068238 12615 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.071664 12796 raft_consensus.cc:2410] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:28.072232 14153 consensus_queue.cc:1059] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260202 15:08:28.108052 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:28.108981 12726 raft_consensus.cc:2393] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:27.999537 13358 raft_consensus.cc:2955] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:28.078114 12798 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:28.050627 14198 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260202 15:08:28.078770 14395 log.cc:927] Time spent T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20260202 15:08:28.080996 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.082062 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.086674 14316 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:28.091717 14227 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.091544 14276 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:28.093700 14185 consensus_meta.cc:287] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.656s user 0.009s sys 0.006s
I20260202 15:08:28.095381 14258 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.096562 14370 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:28.097369 12646 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:28.126679 14058 raft_consensus.cc:697] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:28.097517 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:28.127087 14260 consensus_meta.cc:287] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.515s user 0.006s sys 0.004s
I20260202 15:08:28.113727 14373 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.052122 14215 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:28.127552 14058 consensus_queue.cc:237] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:28.128654 14331 consensus_meta.cc:287] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.509s user 0.003s sys 0.005s
I20260202 15:08:28.129070 14331 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.101694 13360 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:28.131192 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:28.132061 13360 raft_consensus.cc:2410] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:28.102165 14374 consensus_queue.cc:237] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.103386 12722 raft_consensus.cc:2393] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:28.065310 14396 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260202 15:08:28.049942 14208 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:28.106171 12723 consensus_meta.cc:287] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.636s user 0.007s sys 0.012s
I20260202 15:08:28.118590 14316 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:28.036573 14187 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
W20260202 15:08:28.073124 14176 consensus_meta.cc:287] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.640s user 0.018s sys 0.011s
I20260202 15:08:28.109951 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.111765 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:28.112109 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 43937.9us
I20260202 15:08:28.111801 12721 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:28.051126 14252 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260202 15:08:28.117283 12645 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:28.121827 14168 consensus_meta.cc:287] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.780s user 0.007s sys 0.005s
W20260202 15:08:28.110294 14327 consensus_meta.cc:287] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.705s user 0.012s sys 0.009s
W20260202 15:08:28.116416 14338 consensus_meta.cc:287] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.506s user 0.010s sys 0.003s
I20260202 15:08:28.108227 14397 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:28.140808 12614 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.113440 14198 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:28.049306 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:27.548184 12792 rpcz_store.cc:276] 0202 15:08:25.225845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:25.226788 (+ 943us) service_pool.cc:225] Handling call
0202 15:08:25.578280 (+351492us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:25.578674 (+ 394us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:25.583390 (+ 4716us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:25.583865 (+ 475us) log.cc:844] Serialized 26 byte log entry
0202 15:08:26.957343 (+1373478us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:26.963089 (+ 5746us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:26.969808 (+ 6719us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:26.972929 (+ 3121us) raft_consensus.cc:1685] finished
0202 15:08:26.972944 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:27.423760 (+450816us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54002eb650. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:27.427193 (+ 3433us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1326,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"spinlock_wait_cycles":633304960,"thread_start_us":1357955,"threads_started":1,"wal-append.queue_time_us":910807,"wal-append.run_cpu_time_us":8953,"wal-append.run_wall_time_us":1149414}
I20260202 15:08:28.149257 12723 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:28.194278 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1281) took 1018 ms. Trace:
I20260202 15:08:28.194703 12723 rpcz_store.cc:276] 0202 15:08:27.176213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.339845 (+163632us) service_pool.cc:225] Handling call
0202 15:08:28.194214 (+854369us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:28.195822 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.196398 12723 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:28.106719 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:28.121053 14158 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:28.123011 14192 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:28.116794 12764 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
W20260202 15:08:28.157292 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 394791us
I20260202 15:08:28.216166 12763 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.217392 12763 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.218036 14400 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:28.218353 14400 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:28.156354 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:28.130080 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.216092 12723 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:28.219461 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:28.219518 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.219682 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.160073 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.220582 12723 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:28.220984 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:28.162868 13362 raft_consensus.cc:2410] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:28.221434 12615 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.164038 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:28.155252 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 345685us
I20260202 15:08:28.222440 13328 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.222674 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.223335 12726 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 4
W20260202 15:08:28.151310 14199 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.000s sys 0.018s
W20260202 15:08:28.229034 14199 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s user 0.000s sys 0.019s
I20260202 15:08:28.229413 14176 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:28.229877 14193 consensus_meta.cc:287] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.847s user 0.016s sys 0.031s
I20260202 15:08:28.230190 14193 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.173811 12645 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.190126 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.130316 14185 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.170328 14397 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:28.237753 14397 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:28.238675 14185 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.127553 14260 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.240484 14260 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.201133 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.241772 13363 raft_consensus.cc:1240] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:28.244025 14229 consensus_queue.cc:1059] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:28.244988 14229 raft_consensus.cc:3055] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:28.245294 14229 raft_consensus.cc:740] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:28.234553 14259 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.170838 14168 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.248309 14259 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.248749 14168 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.248905 14185 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:28.249413 14185 raft_consensus.cc:697] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:28.159927 14220 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260202 15:08:28.223697 13328 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:28.250119 14185 consensus_queue.cc:237] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.252632 13328 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.253139 14193 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.254137 14370 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:28.254532 14370 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:28.170498 14327 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.226143 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.154663 14316 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.188311 14338 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.171069 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 641) took 1235 ms. Trace:
I20260202 15:08:28.231539 14368 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.191919 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:28.187875 14232 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:28.222126 14192 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:28.235102 14153 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:28.235190 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 240) took 1001 ms. Trace:
W20260202 15:08:28.169773 14202 consensus_meta.cc:287] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.554s user 0.010s sys 0.001s
I20260202 15:08:28.247066 14199 raft_consensus.cc:3055] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:28.246094 14229 consensus_queue.cc:260] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.254930 14370 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:28.212329 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 179936us
I20260202 15:08:28.187866 14198 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:28.218410 12763 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.257436 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:28.248785 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 132070us
I20260202 15:08:28.273237 14192 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:28.214349 14248 consensus_meta.cc:287] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.858s user 0.009s sys 0.030s
W20260202 15:08:28.224324 14293 consensus_meta.cc:287] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.769s user 0.006s sys 0.004s
I20260202 15:08:28.203207 14158 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:28.272485 12647 rpcz_store.cc:276] 0202 15:08:27.234294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.551335 (+317041us) service_pool.cc:225] Handling call
0202 15:08:27.551630 (+ 295us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:27.571255 (+ 19625us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:27.571284 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:27.583520 (+ 12236us) log.cc:844] Serialized 26 byte log entry
0202 15:08:27.997854 (+414334us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:27.997902 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.998020 (+ 118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:28.234976 (+236956us) raft_consensus.cc:1685] finished
0202 15:08:28.234999 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:28.235135 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60878,"raft.queue_time_us":19464,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":15379,"thread_start_us":397246,"threads_started":1,"wal-append.queue_time_us":563006}
I20260202 15:08:28.279841 14153 raft_consensus.cc:697] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:28.200340 14342 raft_consensus.cc:2764] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260202 15:08:28.233667 12726 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:28.306439 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.307528 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.308599 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.218724 14400 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:28.267882 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:28.213246 12717 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 1.
I20260202 15:08:28.259361 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.284092 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.204591 14292 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.287535 12798 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:28.154181 14208 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.298766 14248 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3.
I20260202 15:08:28.272395 14319 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:28.288093 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 255437us
I20260202 15:08:28.303597 14407 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:28.334160 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:28.272858 14276 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.337611 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.258926 14329 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:28.280340 14190 consensus_meta.cc:287] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.898s user 0.010s sys 0.011s
I20260202 15:08:28.297081 13328 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:28.298071 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 281305us
I20260202 15:08:28.265805 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:28.279579 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.345141 12722 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:28.345291 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.279578 14354 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.278980 14199 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:28.277621 14232 raft_consensus.cc:697] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:28.273018 12720 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:28.310252 12615 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.274909 14202 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.316883 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.318863 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.323408 14399 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:28.303021 14158 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:28.291113 12764 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:28.321918 12726 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:28.303414 14293 raft_consensus.cc:2749] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:28.304754 14229 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.324916 12717 raft_consensus.cc:1275] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:28.325268 14215 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.325894 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.326646 13362 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260202 15:08:28.327378 14153 consensus_queue.cc:237] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.335566 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.336114 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:28.336305 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.336872 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.341814 12690 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.342984 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:28.343182 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.397883 14399 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:28.418061 12688 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:28.359387 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 62388.6us
I20260202 15:08:28.394106 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:28.423799 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 82060.7us
I20260202 15:08:28.394306 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.294250 14331 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:28.396409 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 86256.8us
I20260202 15:08:28.423816 13328 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:28.427798 12615 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.428937 12690 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.405481 12642 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:28.409564 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:28.410931 13360 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:28.347714 12800 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
W20260202 15:08:28.407748 14287 consensus_meta.cc:287] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.569s user 0.010s sys 0.002s
I20260202 15:08:28.401019 14319 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:28.344000 14193 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
W20260202 15:08:28.449019 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 200120us
I20260202 15:08:28.387305 14190 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.458420 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.439098 12688 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.431167 13328 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.447536 12647 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260202 15:08:28.425052 14338 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.437335 12724 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:28.463160 12647 pending_rounds.cc:85] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 4
I20260202 15:08:28.420095 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.468804 13328 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.467840 12690 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.264050 13359 rpcz_store.cc:276] 0202 15:08:26.935266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.441358 (+506092us) service_pool.cc:225] Handling call
0202 15:08:27.441636 (+ 278us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:27.442419 (+ 783us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:27.442466 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:27.443092 (+ 626us) log.cc:844] Serialized 26 byte log entry
0202 15:08:27.675153 (+232061us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:27.689619 (+ 14466us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:27.696289 (+ 6670us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:28.158412 (+462123us) raft_consensus.cc:1685] finished
0202 15:08:28.158738 (+ 326us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:28.168620 (+ 9882us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":78,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":196709,"threads_started":1,"wal-append.queue_time_us":403767}
I20260202 15:08:28.422508 12720 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1"
dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:55526
I20260202 15:08:28.466960 12647 pending_rounds.cc:107] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260202 15:08:28.475836 12647 raft_consensus.cc:2889] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:28.435930 14327 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.449303 14276 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:28.448596 14398 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:28.426406 14399 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:28.435737 14368 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:28.439981 14150 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:28.443650 14354 raft_consensus.cc:2764] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260202 15:08:28.471763 14319 raft_consensus.cc:697] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:28.494160 12796 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:28.475534 14179 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:28.475425 14350 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:28.482014 14259 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:28.456840 14341 consensus_queue.cc:1048] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260202 15:08:28.486280 13355 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:28.484990 14276 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:28.435276 14199 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:28.497849 14408 log.cc:927] Time spent T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.208s user 0.000s sys 0.001s
I20260202 15:08:28.491705 14150 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:28.450101 14287 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.499239 14259 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:28.506947 14276 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:28.524561 14190 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:28.516209 14208 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.525475 14287 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.520633 14199 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:28.516570 14403 log.cc:927] Time spent T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.204s user 0.000s sys 0.001s
I20260202 15:08:28.526393 14198 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:28.527230 14343 consensus_meta.cc:287] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.798s user 0.012s sys 0.000s
I20260202 15:08:28.484189 14232 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.528517 14259 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.528390 14329 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.533339 14397 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.542826 14397 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:28.549464 14349 consensus_meta.cc:287] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.165s user 0.010s sys 0.015s
I20260202 15:08:28.549811 14349 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.550374 14316 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:28.551219 14414 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260202 15:08:28.552187 14316 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:28.552522 14413 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260202 15:08:28.553776 14415 log.cc:927] Time spent T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260202 15:08:28.554986 14349 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:28.556792 14268 consensus_meta.cc:287] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.730s user 0.003s sys 0.007s
I20260202 15:08:28.533439 14319 consensus_queue.cc:237] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.558488 14268 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.561390 14370 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.562249 14268 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.537236 14343 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.563093 14370 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:28.564513 14343 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.435199 12646 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:28.602711 12722 raft_consensus.cc:1240] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:28.593140 12720 raft_consensus.cc:606] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:28.603333 12720 raft_consensus.cc:614] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260202 15:08:28.603632 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 223) took 1341 ms. Trace:
W20260202 15:08:28.596678 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 128759us
I20260202 15:08:28.597268 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1269) took 1741 ms. Trace:
W20260202 15:08:28.601029 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 161851us
I20260202 15:08:28.610996 12721 rpcz_store.cc:276] 0202 15:08:26.855834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:26.927498 (+ 71664us) service_pool.cc:225] Handling call
0202 15:08:28.596722 (+1669224us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400145854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:28.596950 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2512878464}
I20260202 15:08:28.604154 12718 rpcz_store.cc:276] 0202 15:08:27.262010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.441367 (+179357us) service_pool.cc:225] Handling call
0202 15:08:27.622931 (+181564us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:27.627556 (+ 4625us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:27.627586 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:27.627995 (+ 409us) log.cc:844] Serialized 26 byte log entry
0202 15:08:28.149830 (+521835us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:28.149888 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:28.149960 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:28.602002 (+452042us) raft_consensus.cc:1685] finished
0202 15:08:28.602037 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:28.603196 (+ 1159us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b5400250dd0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:28.603442 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":214761,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":11460,"spinlock_wait_cycles":416925056,"thread_start_us":453257,"threads_started":1,"wal-append.queue_time_us":604520}
W20260202 15:08:28.617219 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:28.618036 12719 raft_consensus.cc:2393] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:28.617148 12725 raft_consensus.cc:2393] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:28.615736 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.618189 12615 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.614246 12688 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.618673 12690 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:28.619573 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 401005us
W20260202 15:08:28.489717 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 184545us
I20260202 15:08:28.506537 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.514119 14150 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.615109 12720 raft_consensus.cc:2955] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:28.533335 14202 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.713639 12646 raft_consensus.cc:2393] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:28.713757 14101 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:28.714514 14287 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:28.715013 14287 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:28.715463 14287 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:28.552237 14400 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:28.619769 12721 raft_consensus.cc:1240] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:28.557580 14276 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.725311 14363 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:28.474207 13328 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.728184 12796 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:28.734130 12797 raft_consensus.cc:2393] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:28.736169 13359 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:28.627959 12799 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:28.536898 14215 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.627337 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 452) took 1341 ms. Trace:
W20260202 15:08:28.622046 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 330844us
I20260202 15:08:28.639385 13357 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:28.657039 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.657948 14193 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:28.635470 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 451) took 1353 ms. Trace:
I20260202 15:08:28.665393 14327 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:28.660933 14209 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260202 15:08:28.739128 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 121171us
I20260202 15:08:28.741926 14168 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:28.744836 14168 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:28.745334 14168 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:28.745879 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:28.747299 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 128909us
W20260202 15:08:28.748246 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 274141us
I20260202 15:08:28.738646 13361 rpcz_store.cc:276] 0202 15:08:27.286019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.060392 (+774373us) service_pool.cc:225] Handling call
0202 15:08:28.087967 (+ 27575us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:28.088274 (+ 307us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:28.088297 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:28.089193 (+ 896us) log.cc:844] Serialized 564 byte log entry
0202 15:08:28.303504 (+214311us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:28.303541 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:28.303600 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:28.626579 (+322979us) raft_consensus.cc:1685] finished
0202 15:08:28.626614 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:28.626962 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":213935,"threads_started":1,"wal-append.queue_time_us":354977}
I20260202 15:08:28.749022 12690 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.742945 14398 raft_consensus.cc:2749] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:28.750566 14350 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:28.750976 14276 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.750973 14350 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:28.753882 13328 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.600s (SENT) [suppressed 9 similar messages]
I20260202 15:08:28.754279 12763 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.718281 14192 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.742738 13355 rpcz_store.cc:276] 0202 15:08:27.281519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.946203 (+664684us) service_pool.cc:225] Handling call
0202 15:08:27.988426 (+ 42223us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:28.505322 (+516896us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:28.634826 (+129504us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b5400500354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:28.635257 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1115857280}
W20260202 15:08:28.741917 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 130372us
I20260202 15:08:28.755620 14202 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:28.759224 12688 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.759279 14202 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.749835 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:28.745911 13356 consensus_meta.cc:287] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.298s user 0.013s sys 0.017s
I20260202 15:08:28.763088 13356 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:28.763497 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:28.753093 14320 consensus_meta.cc:287] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.880s user 0.013s sys 0.002s
I20260202 15:08:28.763691 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 454) took 1414 ms. Trace:
I20260202 15:08:28.763470 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.764312 14171 raft_consensus.cc:2764] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260202 15:08:28.781471 14209 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260202 15:08:28.765297 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.783231 13363 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:28.766019 14176 raft_consensus.cc:2749] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:28.757558 14343 consensus_queue.cc:1059] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:08:28.787220 12688 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:28.788113 14343 raft_consensus.cc:3055] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:28.788475 12765 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.740890 14193 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:28.789393 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 51555us
I20260202 15:08:28.789482 14193 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.790088 12765 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:28.790755 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:28.790894 14309 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:28.791196 14190 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:28.791684 14190 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:28.762519 14202 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.753907 14350 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.771220 14342 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:28.776284 14254 consensus_queue.cc:1059] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:08:28.777225 14400 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:28.780969 14354 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:28.781539 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.796562 13361 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:28.797305 14399 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.788517 14343 raft_consensus.cc:740] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:28.798897 12764 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.799166 14343 consensus_queue.cc:260] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.800567 14343 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.800921 13362 raft_consensus.cc:2368] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5.
I20260202 15:08:28.803419 12765 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:28.803454 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.804347 14400 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:28.804828 14400 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:28.805303 14400 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.805402 12765 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:28.806140 12645 raft_consensus.cc:1240] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:28.792117 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 457) took 1231 ms. Trace:
I20260202 15:08:28.806916 13360 rpcz_store.cc:276] 0202 15:08:27.560628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.147697 (+587069us) service_pool.cc:225] Handling call
0202 15:08:28.147968 (+ 271us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:28.431626 (+283658us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:28.791644 (+360018us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b540068ef54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:28.791930 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":604223872}
I20260202 15:08:28.809104 14309 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:28.809672 14309 raft_consensus.cc:697] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:28.810042 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.792115 14190 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:28.811187 14220 consensus_queue.cc:1059] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260202 15:08:28.813920 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:28.814821 14309 consensus_queue.cc:237] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.815575 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 450) took 1540 ms. Trace:
I20260202 15:08:28.780089 13356 rpcz_store.cc:276] 0202 15:08:27.349430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.391260 (+ 41830us) service_pool.cc:225] Handling call
0202 15:08:28.763610 (+1372350us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:28.816020 13354 rpcz_store.cc:276] 0202 15:08:27.274652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.885791 (+611139us) service_pool.cc:225] Handling call
0202 15:08:27.942572 (+ 56781us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:28.013899 (+ 71327us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:28.024562 (+ 10663us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:28.096815 (+ 72253us) log.cc:844] Serialized 26 byte log entry
0202 15:08:28.319735 (+222920us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:28.502053 (+182318us) log.cc:844] Serialized 14 byte log entry
0202 15:08:28.627942 (+125889us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:28.635402 (+ 7460us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b5400640354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:28.635462 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:28.780712 (+145250us) spinlock_profiling.cc:243] Waited 4.67 ms on lock 0x7b1800140da0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea4b12d3 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad 0000000000379254 0000000000379099 00007f24ea49469a
0202 15:08:28.815228 (+ 34516us) raft_consensus.cc:1685] finished
0202 15:08:28.815260 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:28.815413 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":217952,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":155025,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":25486,"spinlock_wait_cycles":122493696,"thread_start_us":380039,"threads_started":2,"wal-append.queue_time_us":400356}
I20260202 15:08:28.818116 14292 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:28.814834 14320 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.819343 12642 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:28.819470 14399 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:28.496611 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:28.797808 13358 consensus_meta.cc:287] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.586s user 0.008s sys 0.004s
I20260202 15:08:28.820161 14320 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:28.824801 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 53278us
I20260202 15:08:28.827179 12643 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:28.828527 14198 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.829885 14386 raft_consensus.cc:3055] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:28.830265 14386 raft_consensus.cc:740] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:28.826900 13363 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:28.831216 14386 consensus_queue.cc:260] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.831352 14354 raft_consensus.cc:697] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:28.835577 14386 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:28.836663 14354 consensus_queue.cc:237] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:28.838836 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 51837.4us
I20260202 15:08:28.844730 14345 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:28.845255 14345 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:28.845917 14345 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:28.829111 14192 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.850690 12641 raft_consensus.cc:1240] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:28.851078 14399 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260202 15:08:28.867300 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 57895.5us
I20260202 15:08:28.868794 12690 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:28.870975 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 64616.8us
I20260202 15:08:28.877425 12641 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:28.872196 14150 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:28.520087 14155 consensus_queue.cc:1175] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "66b8c715b9e6479e9218b08bd3156380" responder_term: 4 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: [3.6-3.6]" } } last_received_current_leader { term: 0 index: 0 } }
I20260202 15:08:28.894971 14268 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:28.895462 14268 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:28.895849 14268 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:28.882344 14215 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:28.877736 13355 raft_consensus.cc:2393] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 2: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:28.907222 14192 consensus_queue.cc:1059] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:28.907706 12649 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:28.908228 14192 consensus_queue.cc:1243] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260202 15:08:28.912988 14158 consensus_meta.cc:287] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.502s user 0.008s sys 0.003s
I20260202 15:08:28.913434 14158 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:28.914870 14411 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:28.915550 14411 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:28.903724 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 149869us
I20260202 15:08:28.920691 13361 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:28.852312 14168 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.927846 13362 raft_consensus.cc:1240] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260202 15:08:28.928776 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.929728 12726 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:28.929603 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.930842 12792 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:28.932564 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:28.933835 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:28.870249 14292 consensus_queue.cc:1059] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260202 15:08:28.936204 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:28.937968 12647 raft_consensus.cc:1240] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:28.938030 14193 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.938895 14168 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.940236 14193 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.944775 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:28.945564 12722 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:28.946077 13361 raft_consensus.cc:2393] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:28.856979 14327 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.948249 14327 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.948927 13361 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:28.952373 14233 raft_consensus.cc:3055] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:28.952732 14268 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:28.954870 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:28.917346 14287 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.952759 14233 raft_consensus.cc:740] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:29.237244 14158 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:28.954701 14268 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:29.446833 14400 consensus_meta.cc:287] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.641s user 0.010s sys 0.000s
I20260202 15:08:28.956261 14190 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:28.963884 14350 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:29.471256 14343 consensus_meta.cc:287] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.670s user 0.006s sys 0.005s
I20260202 15:08:28.960639 14259 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:08:29.468886 14309 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.002s sys 0.024s
W20260202 15:08:29.564569 14309 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.002s sys 0.024s
I20260202 15:08:29.570617 14400 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.589831 14400 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:29.570219 14411 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:29.631717 14345 consensus_meta.cc:287] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.785s user 0.012s sys 0.000s
I20260202 15:08:29.639374 14345 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.641156 14345 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:28.958129 12792 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:29.643983 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1680) took 1643 ms. Trace:
I20260202 15:08:29.644420 12792 rpcz_store.cc:276] 0202 15:08:28.000043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.444209 (+444166us) service_pool.cc:225] Handling call
0202 15:08:29.643859 (+1199650us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:29.643221 14287 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:29.642446 14259 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:29.647686 14190 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:29.568452 14233 consensus_queue.cc:260] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.666837 14233 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:29.668592 14306 raft_consensus.cc:3055] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:29.669020 14306 raft_consensus.cc:740] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:29.675206 14350 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:29.677150 14354 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.053s sys 0.108s
W20260202 15:08:29.662544 14386 consensus_meta.cc:287] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.826s user 0.004s sys 0.005s
I20260202 15:08:29.681015 14293 raft_consensus.cc:3055] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:29.670014 14306 consensus_queue.cc:260] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:29.678310 14354 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.054s sys 0.108s
I20260202 15:08:29.681474 14293 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:29.697419 14306 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:29.707167 14293 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.722772 14293 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:29.735033 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:29.735828 12798 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:29.749521 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:29.750247 12725 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:29.760016 12798 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
W20260202 15:08:29.754554 14258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.818s user 0.023s sys 0.036s
I20260202 15:08:29.762440 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1679) took 1760 ms. Trace:
I20260202 15:08:28.957722 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:29.762346 14258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.825s user 0.023s sys 0.036s
I20260202 15:08:29.763325 12795 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:29.763088 12798 rpcz_store.cc:276] 0202 15:08:27.999811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:29.734468 (+1734657us) service_pool.cc:225] Handling call
0202 15:08:29.760581 (+ 26113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:29.766594 12725 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:29.774739 12795 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:29.782225 12725 raft_consensus.cc:1240] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260202 15:08:29.782071 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 239773us
I20260202 15:08:29.783617 12722 raft_consensus.cc:1240] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:29.785360 14233 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:29.786626 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:29.788156 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:29.788605 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:29.789664 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:29.790382 12722 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:29.790052 14233 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.792348 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:29.790532 14306 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:29.793609 14306 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.793092 12725 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:29.795207 14354 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:29.795797 14354 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.794152 14233 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:29.797405 14354 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:29.799032 14306 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:29.811923 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:29.816900 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 973423us
I20260202 15:08:29.818837 14258 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260202 15:08:29.822186 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:29.823192 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:29.823786 14373 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:29.824225 14354 consensus_queue.cc:1059] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:29.824257 14373 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.825482 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:29.826079 12725 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:29.826946 14373 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:29.827507 14259 raft_consensus.cc:3055] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:29.827857 14259 raft_consensus.cc:740] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:29.828567 12690 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:29.828689 14259 consensus_queue.cc:260] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.829319 14354 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:29.829866 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:29.829811 14259 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:29.834530 12722 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:29.835839 12722 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:28.956974 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:29.837651 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:29.838405 12690 connection.cc:232] RPC call timeout handler was delayed by -1.1170642510001016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260202 15:08:29.839748 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:29.840248 12722 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:29.840966 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:29.841692 12722 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:29.842284 14373 consensus_queue.cc:1059] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:29.842984 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:29.844144 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:29.844707 12649 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:29.844926 12725 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:29.846096 12725 raft_consensus.cc:1240] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:29.849036 12721 raft_consensus.cc:1240] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:29.854887 14373 consensus_queue.cc:1059] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260202 15:08:29.857379 12797 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1678) took 1857 ms (client timeout 2044 ms). Trace:
W20260202 15:08:29.857846 12797 rpcz_store.cc:269] 0202 15:08:27.999584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.219305 (+219721us) service_pool.cc:225] Handling call
0202 15:08:29.857112 (+1637807us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b54001cc0d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:29.857308 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1158990592}
I20260202 15:08:29.861754 12650 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:29.866158 14452 raft_consensus.cc:3055] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:29.866593 14452 raft_consensus.cc:740] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:29.874051 14452 consensus_queue.cc:260] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.876024 14452 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:29.877349 12724 raft_consensus.cc:1240] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20260202 15:08:29.866537 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:29.872488 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:29.885056 14452 raft_consensus.cc:3055] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:29.885630 14452 raft_consensus.cc:740] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:29.886797 14452 consensus_queue.cc:260] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.896948 12649 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:29.899412 12718 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
W20260202 15:08:29.904071 12799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 235) took 2151 ms (client timeout 2015 ms). Trace:
I20260202 15:08:29.904243 14287 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:29.904769 14287 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.906756 14287 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:29.904109 14354 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:29.907680 14354 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.908334 14319 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:29.908762 14319 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.909484 14354 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:29.904477 12799 rpcz_store.cc:269] 0202 15:08:27.752624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.753378 (+ 754us) service_pool.cc:225] Handling call
0202 15:08:27.861770 (+108392us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:29.903852 (+2042082us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b5400372dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:29.904002 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1576458752}
I20260202 15:08:28.957295 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:29.914988 14319 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:29.915635 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 73602us
I20260202 15:08:29.918339 14319 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:29.919925 14319 consensus_queue.cc:1059] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:29.920728 14306 raft_consensus.cc:3055] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:29.921037 14306 raft_consensus.cc:740] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:29.921089 12615 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:29.921785 14319 raft_consensus.cc:2804] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:29.922137 14319 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:29.921900 14306 consensus_queue.cc:260] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.922465 14319 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:29.923084 14306 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:29.924942 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 460) took 1028 ms. Trace:
I20260202 15:08:29.925326 13355 rpcz_store.cc:276] 0202 15:08:28.896572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.933536 (+ 36964us) service_pool.cc:225] Handling call
0202 15:08:29.924859 (+991323us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:29.926440 14363 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:29.926815 14363 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:29.929039 12615 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:29.929870 14450 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:29.931773 14319 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.933482 14319 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:29.934365 14345 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:29.934875 14345 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.935995 14309 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:29.936720 14345 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:29.937172 14345 consensus_queue.cc:1059] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:29.938037 14345 raft_consensus.cc:3055] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:29.938167 14150 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:29.938361 14345 raft_consensus.cc:740] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:29.936496 14309 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.939898 14309 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:29.940678 14345 consensus_queue.cc:260] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:29.941884 14345 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:29.942003 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:29.942516 12648 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:29.934152 14451 consensus_queue.cc:1059] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260202 15:08:29.948551 14319 consensus_queue.cc:1059] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:29.951509 14456 raft_consensus.cc:3055] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:29.951804 14456 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:29.952703 14456 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.954275 14456 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:29.959376 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 225) took 2696 ms (client timeout 2230 ms). Trace:
W20260202 15:08:29.959785 12719 rpcz_store.cc:269] 0202 15:08:27.262455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.362415 (+ 99960us) service_pool.cc:225] Handling call
0202 15:08:29.959112 (+2596697us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540009f1d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:29.959274 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2390238592}
I20260202 15:08:29.966106 12763 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:29.966948 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:29.967454 12723 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:29.969586 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:29.969513 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:29.970304 12645 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:29.970412 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:29.976109 14350 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:29.976678 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:29.977123 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:29.978181 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 466) took 1015 ms. Trace:
I20260202 15:08:29.976713 14350 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.978528 13355 rpcz_store.cc:276] 0202 15:08:28.962934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:29.969357 (+1006423us) service_pool.cc:225] Handling call
0202 15:08:29.978126 (+ 8769us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:29.980010 14350 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:28.968124 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:29.977794 13356 raft_consensus.cc:1240] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:29.977792 12650 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:29.980583 12689 leader_election.cc:400] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5.
I20260202 15:08:29.983558 12689 leader_election.cc:403] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260202 15:08:29.984037 14287 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5.
I20260202 15:08:29.984457 12689 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:29.985275 14287 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election won for term 3
W20260202 15:08:29.985562 13358 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.098s user 0.007s sys 0.019s
W20260202 15:08:29.985919 13358 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.099s user 0.007s sys 0.019s
I20260202 15:08:29.985785 14287 raft_consensus.cc:697] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:29.986296 13357 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 448) took 2717 ms (client timeout 1752 ms). Trace:
W20260202 15:08:29.986675 13357 rpcz_store.cc:269] 0202 15:08:27.268844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.289450 (+ 20606us) service_pool.cc:225] Handling call
0202 15:08:29.986085 (+2696635us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54000d34d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:29.986221 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2778792832}
I20260202 15:08:29.987591 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 442) took 2957 ms. Trace:
I20260202 15:08:29.987020 14287 consensus_queue.cc:237] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.980597 14350 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:29.987876 12723 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:29.988317 14350 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:29.988721 13362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 464) took 1536 ms. Trace:
I20260202 15:08:29.989563 13362 rpcz_store.cc:276] 0202 15:08:28.452248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.801596 (+349348us) service_pool.cc:225] Handling call
0202 15:08:28.921241 (+119645us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:28.928283 (+ 7042us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:29.988595 (+1060312us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400483350. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:29.988679 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":274201344}
I20260202 15:08:29.990339 14350 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:29.990842 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:29.991609 13362 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:29.991750 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:29.992635 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:29.993924 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:29.982537 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 466) took 1280 ms. Trace:
I20260202 15:08:29.985350 12689 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:29.987812 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:29.995887 12717 raft_consensus.cc:1240] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:30.005998 14176 consensus_queue.cc:1059] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.011s
I20260202 15:08:30.008968 13354 raft_consensus.cc:1240] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:30.017366 14176 raft_consensus.cc:3055] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:30.001021 14258 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:30.017660 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 468) took 1200 ms. Trace:
I20260202 15:08:29.994724 13362 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:30.018080 13354 rpcz_store.cc:276] 0202 15:08:28.816830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:30.008471 (+1191641us) service_pool.cc:225] Handling call
0202 15:08:30.008777 (+ 306us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:30.017430 (+ 8653us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:30.017593 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:30.019482 13362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 465) took 1057 ms. Trace:
I20260202 15:08:30.019757 13362 rpcz_store.cc:276] 0202 15:08:28.961839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:30.018841 (+1057002us) service_pool.cc:225] Handling call
0202 15:08:30.019092 (+ 251us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:30.019235 (+ 143us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:30.019256 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:30.019289 (+ 33us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:30.019319 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:30.019361 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:30.019438 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:30.012539 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.020848 13354 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:30.021160 13357 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:30.022435 13354 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:30.023793 13354 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:30.025305 13354 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:30.025908 13357 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:30.014364 14452 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:30.030618 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.007361 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.031270 13355 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:30.031630 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.032501 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.033051 12718 raft_consensus.cc:2393] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:30.033658 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.034073 13326 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.034850 14350 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:30.035071 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.035363 14350 raft_consensus.cc:697] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:30.035991 12721 raft_consensus.cc:2410] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:30.036319 14350 consensus_queue.cc:237] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.036733 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.037693 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.038326 12721 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:30.039151 12725 raft_consensus.cc:1240] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:30.042369 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.042652 14363 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.043640 12644 raft_consensus.cc:2368] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4.
I20260202 15:08:30.044103 12641 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260202 15:08:30.044205 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.044567 14150 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:30.044433 12641 pending_rounds.cc:85] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 4
I20260202 15:08:30.045111 12614 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.045516 12641 pending_rounds.cc:107] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260202 15:08:30.045871 12641 raft_consensus.cc:2889] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:30.045923 14150 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:29.995555 13356 rpcz_store.cc:276] 0202 15:08:28.726065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.822637 (+ 96572us) service_pool.cc:225] Handling call
0202 15:08:28.923467 (+100830us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:29.981211 (+1057744us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:29.982279 (+ 1068us) spinlock_profiling.cc:243] Waited 90.1 ms on lock 0x7b540041bed0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:29.982437 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207335808}
I20260202 15:08:29.987923 13358 rpcz_store.cc:276] 0202 15:08:27.030205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:27.868624 (+838419us) service_pool.cc:225] Handling call
0202 15:08:27.868985 (+ 361us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:27.995247 (+126262us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:08:28.886735 (+891488us) log.cc:844] Serialized 14 byte log entry
0202 15:08:29.986355 (+1099620us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:29.986425 (+ 70us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:29.986501 (+ 76us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:29.987372 (+ 871us) spinlock_profiling.cc:243] Waited 92.1 ms on lock 0x7b5400165dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:29.987411 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:29.987535 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4536,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":94,"spinlock_wait_cycles":211899136,"thread_start_us":1099197,"threads_started":1,"wal-append.queue_time_us":757270}
I20260202 15:08:30.049428 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.050998 14199 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:30.051438 14199 raft_consensus.cc:697] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:30.052243 14199 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:30.054754 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 251) took 1598 ms (client timeout 1553 ms). Trace:
W20260202 15:08:30.055193 12642 rpcz_store.cc:269] 0202 15:08:28.457872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.458194 (+ 322us) service_pool.cc:225] Handling call
0202 15:08:30.054514 (+1596320us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b540008a2d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:30.054660 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":819876736}
I20260202 15:08:30.057332 14309 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:30.057792 14309 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.059500 14309 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.061724 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 465) took 1431 ms. Trace:
I20260202 15:08:30.062100 13360 rpcz_store.cc:276] 0202 15:08:28.630436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.823921 (+193485us) service_pool.cc:225] Handling call
0202 15:08:28.876619 (+ 52698us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:30.061424 (+1184805us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:30.061451 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:30.061495 (+ 44us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:30.061521 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:30.061569 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:30.061667 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:30.067445 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.068668 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.069800 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.070364 12722 raft_consensus.cc:2368] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4.
I20260202 15:08:30.071295 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.071609 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.072750 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.073490 12649 raft_consensus.cc:2410] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260202 15:08:30.076776 12720 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.162s user 0.008s sys 0.011s
I20260202 15:08:30.046619 14190 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:30.077214 12720 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.163s user 0.008s sys 0.011s
I20260202 15:08:30.077462 14190 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:30.077853 12641 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:28.966933 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.078908 12794 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:30.079247 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 242) took 1121 ms. Trace:
I20260202 15:08:30.034516 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.079555 12794 rpcz_store.cc:276] 0202 15:08:28.957899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.966567 (+ 8668us) service_pool.cc:225] Handling call
0202 15:08:30.079194 (+1112627us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:30.079931 12718 raft_consensus.cc:2368] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3.
I20260202 15:08:30.046492 12614 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:29.999444 12800 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:30.081216 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 241) took 1123 ms. Trace:
I20260202 15:08:30.081534 12800 rpcz_store.cc:276] 0202 15:08:28.957544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.967758 (+ 10214us) service_pool.cc:225] Handling call
0202 15:08:30.081164 (+1113406us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:30.082255 14341 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:30.076800 12722 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:30.079321 14190 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.084765 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.087636 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1601) took 1657 ms. Trace:
I20260202 15:08:30.088112 12724 rpcz_store.cc:276] 0202 15:08:28.430207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.616543 (+186336us) service_pool.cc:225] Handling call
0202 15:08:28.620867 (+ 4324us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:29.877767 (+1256900us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:30.087267 (+209500us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540033ac54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:30.087571 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2888274176}
I20260202 15:08:30.017694 14176 raft_consensus.cc:740] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:30.091370 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.092602 12724 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 2: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:30.093590 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.096371 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1592) took 2087 ms. Trace:
I20260202 15:08:30.096781 12717 rpcz_store.cc:276] 0202 15:08:28.009414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.602063 (+592649us) service_pool.cc:225] Handling call
0202 15:08:29.995574 (+1393511us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:30.016659 (+ 21085us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:30.096145 (+ 79486us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b540009f1d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:30.096308 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3203806976}
I20260202 15:08:30.098109 12641 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:30.099088 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:28.963029 13361 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260202 15:08:30.100169 13361 pending_rounds.cc:85] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Aborting all ops after (but not including) 4
I20260202 15:08:30.100404 13361 pending_rounds.cc:107] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260202 15:08:30.100708 13361 raft_consensus.cc:2889] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:30.101423 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 464) took 1140 ms. Trace:
I20260202 15:08:30.101737 13361 rpcz_store.cc:276] 0202 15:08:28.961472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.961990 (+ 518us) service_pool.cc:225] Handling call
0202 15:08:28.962284 (+ 294us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:30.101177 (+1138893us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:30.101367 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:30.011444 12723 raft_consensus.cc:1240] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:30.107038 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.107205 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:30.110271 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.111505 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.112901 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.113979 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:30.118384 13359 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 660) took 1692 ms (client timeout 1618 ms). Trace:
W20260202 15:08:30.118897 13359 rpcz_store.cc:269] 0202 15:08:28.422150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.491007 (+ 68857us) service_pool.cc:225] Handling call
0202 15:08:30.114433 (+1623426us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b5400610154. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:30.114645 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3243392}
I20260202 15:08:30.100324 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.122925 12641 raft_consensus.cc:2393] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 2: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:30.123736 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1595) took 1962 ms. Trace:
I20260202 15:08:30.126715 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.127595 12644 raft_consensus.cc:2393] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:30.126173 12720 rpcz_store.cc:276] 0202 15:08:28.160893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.604486 (+443593us) service_pool.cc:225] Handling call
0202 15:08:28.614187 (+ 9701us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:28.614767 (+ 580us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:08:28.914270 (+299503us) log.cc:844] Serialized 14 byte log entry
0202 15:08:30.123447 (+1209177us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:30.123489 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:30.123530 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:30.123600 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:30.123693 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45810,"raft.queue_time_us":47408,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"thread_start_us":1162958,"threads_started":1,"wal-append.queue_time_us":732267}
I20260202 15:08:30.127530 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.128331 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.128810 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.129894 12646 raft_consensus.cc:2410] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:30.126715 12688 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.131171 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.167469 12643 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
W20260202 15:08:30.167881 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.168326 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.168969 12643 raft_consensus.cc:2393] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:30.134003 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1370) took 1288 ms. Trace:
I20260202 15:08:30.169478 14287 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:30.169570 12647 rpcz_store.cc:276] 0202 15:08:28.846649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:28.846923 (+ 274us) service_pool.cc:225] Handling call
0202 15:08:28.847288 (+ 365us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:28.938364 (+ 91076us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:30.133795 (+1195431us) spinlock_profiling.cc:243] Waited 90.5 ms on lock 0x7b54000794d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:30.133930 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":208140672}
I20260202 15:08:30.169992 14287 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:30.170990 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 44385us
I20260202 15:08:30.144407 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.171559 12800 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:30.171752 14287 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:30.144747 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.172233 14287 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:30.172631 14287 raft_consensus.cc:697] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:30.145783 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.173457 14287 consensus_queue.cc:237] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.145154 14400 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:30.174978 14400 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.176715 14400 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.156258 14411 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:30.179023 14411 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:30.179749 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 179444us
I20260202 15:08:30.180459 13328 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.180757 14411 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.158402 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260202 15:08:30.164542 12641 raft_consensus.cc:1240] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:30.181201 14316 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:30.181492 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 190858us
I20260202 15:08:30.181833 14316 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.090070 14176 consensus_queue.cc:260] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.046540 14150 raft_consensus.cc:697] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:30.172637 12794 raft_consensus.cc:2393] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:30.191044 12644 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:30.187872 12796 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:30.016027 12648 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:30.189986 12614 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.013116 12645 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:30.194967 12688 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:30.197909 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.199888 14373 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:30.200066 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.199956 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.201035 12650 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260202 15:08:30.208760 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.209519 12795 raft_consensus.cc:2368] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:30.215370 14459 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:30.216198 14459 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:30.216781 14459 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.219210 14459 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:30.223922 14459 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:30.224418 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 667) took 1367 ms. Trace:
I20260202 15:08:30.226523 12614 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.227886 12688 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.228169 14450 raft_consensus.cc:697] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:30.231025 14450 consensus_queue.cc:237] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.225214 13363 rpcz_store.cc:276] 0202 15:08:28.857207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:29.993419 (+1136212us) service_pool.cc:225] Handling call
0202 15:08:29.993655 (+ 236us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:29.993836 (+ 181us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:30.009421 (+ 15585us) log.cc:844] Serialized 14 byte log entry
0202 15:08:30.224083 (+214662us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:30.224138 (+ 55us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:30.224190 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:30.224242 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:30.224358 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":14754,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"thread_start_us":229488,"threads_started":2,"wal-append.queue_time_us":214555}
I20260202 15:08:30.237105 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:30.235774 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 55425.1us
I20260202 15:08:30.244755 14455 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:30.245685 14233 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:30.249389 14233 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:30.249892 14233 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:30.246979 14452 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260202 15:08:30.258981 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 50403.3us
I20260202 15:08:30.236989 12646 raft_consensus.cc:1240] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:30.236979 14176 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:30.261365 12763 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.263566 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.225261 14459 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.266148 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.266620 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:30.267829 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 315308us
I20260202 15:08:30.266825 12798 raft_consensus.cc:2393] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:30.237040 14456 consensus_queue.cc:1059] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260202 15:08:30.277189 12765 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.277690 12720 raft_consensus.cc:1240] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:30.279863 12717 raft_consensus.cc:1240] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:30.279670 12547 catalog_manager.cc:5654] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 3, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:08:30.280392 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.281023 13358 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEARNER]: Advancing to term 5
I20260202 15:08:30.263123 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.297595 12547 catalog_manager.cc:5654] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 2 to 4, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 4 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:30.282209 12726 raft_consensus.cc:1240] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:30.282657 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.283036 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:30.284068 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:30.284165 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.284293 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.285123 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.282671 14233 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.286378 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:30.305333 13357 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:30.303712 14293 consensus_queue.cc:1059] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260202 15:08:30.287303 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.306921 14468 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:30.307405 13356 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:30.288372 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:30.263092 14411 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:30.284082 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:30.267330 14400 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:30.308414 14411 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.308846 14400 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:30.308816 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:30.309144 14423 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:30.309384 14400 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:30.309720 14423 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.310218 14411 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:30.289078 12794 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:30.311254 12719 raft_consensus.cc:2393] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 2: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:30.311487 14423 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.312297 14293 raft_consensus.cc:3055] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:30.312173 14476 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260202 15:08:30.312645 14293 raft_consensus.cc:740] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:30.313454 14293 consensus_queue.cc:260] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.288787 14150 consensus_queue.cc:237] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.315091 14293 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:30.315541 14400 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.287276 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.295109 12763 leader_election.cc:400] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4.
I20260202 15:08:30.297405 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.287251 14287 raft_consensus.cc:2749] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:30.281822 12722 raft_consensus.cc:1240] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:30.301553 13363 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:30.302186 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.306080 12547 catalog_manager.cc:5654] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 2 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:30.323621 13361 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:30.319062 12763 leader_election.cc:403] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260202 15:08:30.298286 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.320269 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.320890 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.327822 12796 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:30.321425 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.322054 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.324908 12547 catalog_manager.cc:5654] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 2 to 3, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:30.331754 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.332645 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:30.333859 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 44582.7us
I20260202 15:08:30.335675 12765 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.337065 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.337409 12689 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:30.337685 14319 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:30.338164 12641 raft_consensus.cc:1240] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:30.338091 14319 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.340201 14319 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:30.340381 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.340773 14343 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:30.337409 13361 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:30.342773 13355 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:30.334544 14423 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:30.346921 14474 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:30.347564 14474 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.349303 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.349617 14474 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.350296 14343 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:30.350786 14343 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.352602 12793 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:30.352681 14343 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.353533 13360 raft_consensus.cc:2393] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:08:30.356531 14466 log.cc:927] Time spent T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260202 15:08:30.357069 12647 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:30.359556 14306 raft_consensus.cc:3055] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:30.359913 14306 raft_consensus.cc:740] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:30.360709 14306 consensus_queue.cc:260] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.361865 14306 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:30.363255 14400 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.364363 14459 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:30.366791 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.366859 12763 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.308128 14233 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:30.368036 12763 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.369130 14316 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:30.369563 14190 raft_consensus.cc:2749] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:30.370218 14190 raft_consensus.cc:2764] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260202 15:08:30.370711 14190 consensus_queue.cc:1059] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:30.370875 14345 consensus_queue.cc:1059] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:30.371399 14190 raft_consensus.cc:3055] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:30.371683 14190 raft_consensus.cc:740] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:30.371768 14411 raft_consensus.cc:3055] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:30.372085 14411 raft_consensus.cc:740] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:30.372339 14190 consensus_queue.cc:260] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.372776 14411 consensus_queue.cc:260] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.373421 14190 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:30.374321 14373 raft_consensus.cc:3055] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:30.374683 14373 raft_consensus.cc:740] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:30.375166 12763 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.375520 14373 consensus_queue.cc:260] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.376471 14373 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:30.376998 14468 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:30.377486 14468 raft_consensus.cc:697] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:30.378288 14468 consensus_queue.cc:237] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.380893 12763 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.381752 12765 leader_election.cc:400] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4.
W20260202 15:08:30.381704 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 154194us
W20260202 15:08:30.382364 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 45804.3us
W20260202 15:08:30.383136 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 45779.7us
I20260202 15:08:30.383656 12689 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:30.384889 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 117146us
I20260202 15:08:30.380196 14411 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:30.388845 12765 leader_election.cc:400] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3.
I20260202 15:08:30.389488 12765 leader_election.cc:403] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260202 15:08:30.369717 13358 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEARNER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:08:30.392227 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.334652 12547 catalog_manager.cc:5654] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 2 to 3, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:30.397611 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 63396.7us
I20260202 15:08:30.398859 13328 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.387063 14287 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:30.400049 12614 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:30.409219 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 69027.6us
I20260202 15:08:30.408262 12547 catalog_manager.cc:5654] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:30.394294 14309 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:30.396458 14316 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:30.413306 12795 raft_consensus.cc:1240] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:30.414148 12547 catalog_manager.cc:5654] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 2 to 3, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 3 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:30.415654 12547 catalog_manager.cc:5654] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: config changed from index 4 to 5, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:30.419003 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.409638 14456 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:30.424615 12547 catalog_manager.cc:5654] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 2 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:08:30.405833 14450 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260202 15:08:30.427512 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.427995 12794 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:30.431728 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.414393 14316 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:30.433975 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.437996 12800 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:30.435184 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.435513 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.416323 14176 raft_consensus.cc:2749] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:30.451597 12764 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.451785 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.452267 12796 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
W20260202 15:08:30.437232 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 48612.1us
I20260202 15:08:30.452749 12615 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.452980 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.453123 13328 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.453521 12796 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:30.453970 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.454277 13328 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.454509 12795 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:30.455960 14478 consensus_queue.cc:1059] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260202 15:08:30.456629 14459 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:30.044273 14363 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:30.457324 14459 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:30.457540 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.416427 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.456712 14478 raft_consensus.cc:3055] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:30.455545 13328 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.474176 14468 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:30.475669 14468 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:30.477274 14176 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:30.476750 14468 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.477764 14176 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:30.459523 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 62549.3us
I20260202 15:08:30.421612 14309 raft_consensus.cc:697] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:30.461172 12800 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:30.479666 14176 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.479693 14309 consensus_queue.cc:237] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:08:30.468008 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 57283.8us
I20260202 15:08:30.482539 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:30.485373 12793 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:30.485649 12796 raft_consensus.cc:1240] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:30.486406 13328 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.486722 12793 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:08:30.486833 14481 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:30.487478 14354 consensus_queue.cc:1059] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260202 15:08:30.487848 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 71518.8us
I20260202 15:08:30.488399 14354 raft_consensus.cc:3055] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:30.488741 14354 raft_consensus.cc:740] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:30.489151 12615 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.489497 14354 consensus_queue.cc:260] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.490288 12615 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:30.435220 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.490872 14459 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:30.491473 12791 raft_consensus.cc:2393] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:30.491564 14459 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:30.491734 14199 raft_consensus.cc:2749] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:30.432693 14456 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:30.492398 14199 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:30.493371 13328 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.494103 12764 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.494498 12799 raft_consensus.cc:2393] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:30.486747 12796 raft_consensus.cc:1240] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:30.430048 14400 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3.
I20260202 15:08:30.492447 14456 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:30.495767 12689 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.458748 14316 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:30.405453 14451 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
W20260202 15:08:30.506145 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 113506us
I20260202 15:08:30.508930 12764 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.495223 14354 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:30.513222 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.513360 13361 raft_consensus.cc:1240] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:30.516845 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:30.517062 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 128316us
I20260202 15:08:30.517144 13362 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:30.517405 12648 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:30.472702 14478 raft_consensus.cc:740] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:30.524602 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.525574 12645 raft_consensus.cc:2393] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:30.462517 14268 consensus_queue.cc:1059] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260202 15:08:30.503535 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:30.529515 13328 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.529287 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.534116 13326 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.513576 14363 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:30.594962 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 86082.2us
I20260202 15:08:30.614841 14484 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:30.599244 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 109050us
I20260202 15:08:30.600538 14345 consensus_queue.cc:1059] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
W20260202 15:08:30.607363 14470 log.cc:927] Time spent T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260202 15:08:30.600890 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:30.601661 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 72195.8us
I20260202 15:08:30.628427 12649 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:30.628720 13328 leader_election.cc:400] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:30.510313 14287 raft_consensus.cc:697] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:30.521344 14176 raft_consensus.cc:2749] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260202 15:08:30.603439 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 69393.3us
W20260202 15:08:30.605147 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 87527.7us
W20260202 15:08:30.630126 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.603231 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.607111 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.630946 12650 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:30.606304 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.529763 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:30.615268 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.533108 12642 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:30.535226 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.632893 13355 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:30.599514 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.516566 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.614781 14350 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:30.628983 13328 leader_election.cc:403] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260202 15:08:30.616559 14475 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:30.627086 12792 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:30.640548 12614 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.639417 13326 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.644371 12645 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:30.646404 12615 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.651048 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.616824 14468 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.641925 14350 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:30.641652 14459 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:30.667048 12646 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:30.678081 13361 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260202 15:08:30.657407 14363 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:30.660874 14400 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:30.654551 14475 raft_consensus.cc:697] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:30.668370 14350 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:30.663407 14176 raft_consensus.cc:2749] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:30.669781 14476 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:30.681582 14233 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:30.687635 14363 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:30.666105 14492 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:30.700693 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:30.698226 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 81116.2us
I20260202 15:08:30.708328 12615 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:30.711385 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.711583 12643 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:30.714331 12794 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
W20260202 15:08:30.720677 14485 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260202 15:08:30.721915 14494 consensus_queue.cc:1059] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260202 15:08:30.723198 14233 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:30.724696 14494 consensus_queue.cc:1243] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260202 15:08:30.735307 14475 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:30.719352 14480 log.cc:927] Time spent T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260202 15:08:30.719920 14474 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:30.635226 14478 consensus_queue.cc:260] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.744472 14478 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:30.680128 14459 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.748112 14459 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:30.698799 14476 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:30.702406 14363 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:30.754230 14158 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:30.764765 14158 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.766562 14158 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:30.699381 14482 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.226s
I20260202 15:08:30.693264 14287 consensus_queue.cc:237] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:30.697486 14479 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260202 15:08:30.755939 14498 raft_consensus.cc:3055] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:30.772004 14498 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:30.772717 14498 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:30.773537 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 65310.1us
I20260202 15:08:30.773902 14498 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:30.764153 14456 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.774757 12615 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.776285 14456 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:30.757576 14423 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4.
I20260202 15:08:30.696094 14487 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260202 15:08:30.759824 13360 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:30.780084 12798 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:30.764253 14476 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:30.783638 14482 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:30.779172 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:30.788003 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:30.770349 14363 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:30.780992 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 139205us
W20260202 15:08:30.845455 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.865720 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:30.866438 13360 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:30.781605 14233 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260202 15:08:30.868832 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 47470.3us
W20260202 15:08:30.871970 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 50124us
W20260202 15:08:30.873157 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 345300us
I20260202 15:08:30.878217 12647 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260202 15:08:30.878535 12647 pending_rounds.cc:85] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 4
I20260202 15:08:30.878784 12647 pending_rounds.cc:107] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260202 15:08:30.879014 12647 raft_consensus.cc:2889] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:30.799454 12797 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:30.866940 14450 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:30.884068 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 109429us
I20260202 15:08:30.886798 12765 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.889956 12765 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.870000 13355 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:30.924392 12689 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:30.893172 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:30.929831 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.930125 12647 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:08:30.893477 14494 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:30.896533 14487 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:30.936712 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 320041us
W20260202 15:08:30.932276 14343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.005s sys 0.027s
W20260202 15:08:30.941313 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 290650us
I20260202 15:08:30.944873 12764 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.895290 14190 consensus_queue.cc:1059] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260202 15:08:30.898372 14316 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.944049 12763 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:30.926744 14345 consensus_queue.cc:1059] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
W20260202 15:08:30.944036 14343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.523s user 0.005s sys 0.027s
I20260202 15:08:30.958065 12795 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260202 15:08:30.937556 12646 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
W20260202 15:08:30.960631 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.965631 12764 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.962911 14411 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260202 15:08:30.963569 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:30.967914 12763 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.946846 14500 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:30.969676 12722 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:30.953321 14487 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:30.960865 14486 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260202 15:08:30.972597 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.963966 14350 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:30.981150 14316 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:30.974025 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 345379us
I20260202 15:08:30.975673 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.976190 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:30.976141 14306 raft_consensus.cc:2955] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:30.946523 14494 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260202 15:08:30.979220 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:30.909942 14503 consensus_queue.cc:1059] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260202 15:08:30.983532 13328 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:30.984277 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:30.968299 14411 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:31.009248 14468 raft_consensus.cc:3055] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:31.009523 14411 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:31.009594 14468 raft_consensus.cc:740] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:31.010885 14492 consensus_queue.cc:1059] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:31.011677 14481 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:31.011803 14492 raft_consensus.cc:3055] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:31.012111 14492 raft_consensus.cc:740] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:31.013018 14503 raft_consensus.cc:3055] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:31.013375 14503 raft_consensus.cc:740] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:31.007854 14343 raft_consensus.cc:3055] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:31.014560 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:31.014859 14343 raft_consensus.cc:740] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:31.015296 12646 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:30.983181 14363 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:31.015872 14343 consensus_queue.cc:260] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:31.017236 14343 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:30.967382 14482 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:30.958513 14158 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260202 15:08:31.003048 14512 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:31.003629 12719 raft_consensus.cc:2410] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:31.006930 14487 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:31.006836 14476 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.040716 14494 raft_consensus.cc:2764] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260202 15:08:31.007797 12717 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 2: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:31.014014 12764 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.969198 14500 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:31.019513 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:31.014104 14503 consensus_queue.cc:260] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:31.022918 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 55171.9us
W20260202 15:08:31.021157 13328 connection.cc:232] RPC call timeout handler was delayed by -1.9538821439998628s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
I20260202 15:08:31.061945 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:31.022277 14176 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:31.047103 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:31.010460 14468 consensus_queue.cc:260] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:31.012115 14481 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:31.063169 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 49233.8us
W20260202 15:08:31.118964 13328 connection.cc:232] RPC call timeout handler was delayed by -1.1139563440001439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2903s for a response
I20260202 15:08:31.083413 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:31.124490 12764 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:31.124847 12648 raft_consensus.cc:2393] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:31.062537 12763 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.125706 12764 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:31.111258 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:31.078451 13360 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:31.127768 12724 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:31.049072 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:31.130631 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 68142us
I20260202 15:08:31.131461 12763 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:30.996768 14484 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:31.133322 12763 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.136526 12614 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:31.041213 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:31.138639 14309 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:31.138818 14487 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:31.154002 14487 raft_consensus.cc:697] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:31.154440 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:31.139380 14505 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:31.046344 14512 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.140856 12725 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:31.142409 12717 raft_consensus.cc:3055] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:31.143034 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:31.143666 14363 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
W20260202 15:08:31.144228 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 78997.1us
I20260202 15:08:31.132916 14494 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:31.144584 14510 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:08:31.145321 14423 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260202 15:08:31.093812 13384 debug-util.cc:398] Leaking SignalData structure 0x7b08002c88c0 after lost signal to thread 11454
I20260202 15:08:31.148429 12690 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.061662 14500 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:31.151458 12763 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.107090 14475 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:31.063884 14350 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:31.177309 12763 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.162024 12764 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.079434 14476 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:31.117277 14459 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:31.063935 14502 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:31.043119 14482 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:31.170533 12795 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:08:31.188500 13328 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.043365 14509 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.124s
I20260202 15:08:31.158535 14400 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:31.194288 14350 raft_consensus.cc:2749] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:31.162587 14505 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:31.191669 13354 raft_consensus.cc:2393] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:31.168663 12717 raft_consensus.cc:740] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:31.193616 13362 raft_consensus.cc:2955] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:31.174100 14345 raft_consensus.cc:2749] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:31.189340 14363 raft_consensus.cc:697] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:31.221729 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:31.222569 12798 raft_consensus.cc:2393] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:08:31.224088 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 62166.2us
I20260202 15:08:31.221165 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:31.212366 12719 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:31.238121 12615 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:31.238282 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:31.238346 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:31.238718 14508 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:31.237725 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:31.216253 13384 debug-util.cc:398] Leaking SignalData structure 0x7b08002acba0 after lost signal to thread 12582
W20260202 15:08:31.247335 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 98984.7us
W20260202 15:08:31.252938 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.155005 14487 consensus_queue.cc:237] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:31.212561 14494 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:31.208602 14505 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:31.255136 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 101984us
I20260202 15:08:31.256932 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:31.257731 14451 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:31.257848 12641 raft_consensus.cc:1275] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:31.266315 12615 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:31.266884 12690 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:31.259145 12722 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:08:31.260296 14482 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:31.261394 14492 consensus_queue.cc:260] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.188535 14475 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.194630 14459 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.239863 14481 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:31.264724 13328 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:31.265333 12724 raft_consensus.cc:1275] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260202 15:08:31.271575 12724 pending_rounds.cc:85] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Aborting all ops after (but not including) 1
I20260202 15:08:31.271773 12724 pending_rounds.cc:107] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260202 15:08:31.271963 12724 raft_consensus.cc:2889] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:31.265488 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:31.259021 14512 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:31.269142 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:31.269691 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:31.273030 12795 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:31.266319 14451 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:31.276233 14451 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:31.277346 14503 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:31.277585 14411 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:31.280946 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:31.287226 12689 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:31.287729 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.291579 14475 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:31.270613 14478 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260202 15:08:31.285714 14411 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:31.292169 14176 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:31.295511 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:31.296084 12642 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:31.286062 14190 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260202 15:08:31.300647 14459 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:31.298771 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:31.302947 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:31.303435 12650 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260202 15:08:31.303845 13328 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.272430 14484 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:31.317911 14498 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:31.318992 12689 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.274278 14450 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:31.322116 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:31.326776 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 63670.8us
W20260202 15:08:31.331924 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 77311.8us
I20260202 15:08:31.332679 12690 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:31.301450 14512 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:31.319025 13384 debug-util.cc:398] Leaking SignalData structure 0x7b08003a7fa0 after lost signal to thread 12674
I20260202 15:08:31.334877 13328 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.285575 14451 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:31.324307 14476 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:31.347932 12642 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:31.331708 14268 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:31.351065 12645 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260202 15:08:31.352330 12763 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:31.352805 12690 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.334523 14158 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:31.345788 14492 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:31.360419 12642 raft_consensus.cc:1240] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:31.343598 14287 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:31.304047 14502 consensus_queue.cc:1048] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260202 15:08:31.346500 14450 raft_consensus.cc:697] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
W20260202 15:08:31.373361 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.373188 12615 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:31.324565 14363 consensus_queue.cc:237] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.349053 14484 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:31.376410 13328 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.391440 14495 consensus_queue.cc:1059] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:08:31.393237 14354 raft_consensus.cc:3055] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:31.393703 14354 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:31.379098 13384 debug-util.cc:398] Leaking SignalData structure 0x7b08003f0d60 after lost signal to thread 12750
I20260202 15:08:31.376420 14411 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:31.379283 14455 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:31.394270 14533 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:31.361845 14158 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:31.394758 14268 raft_consensus.cc:2749] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:31.357654 14500 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.440328 12615 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:31.489470 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 49204.2us
W20260202 15:08:31.440711 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 87961.8us
W20260202 15:08:31.440658 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 64322.4us
I20260202 15:08:31.386790 14293 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260202 15:08:31.494156 12690 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:31.490805 13328 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:31.473073 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.432423 14354 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:31.499284 14354 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:31.500207 14535 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:31.499897 12717 consensus_queue.cc:260] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.500682 14535 raft_consensus.cc:697] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:31.501736 12717 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:31.501557 14535 consensus_queue.cc:237] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:31.499315 14538 log.cc:927] Time spent T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260202 15:08:31.504176 14158 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:31.387985 14484 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:31.506230 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 125430us
I20260202 15:08:31.453420 14411 raft_consensus.cc:697] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:31.496523 14482 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:31.511404 12615 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:31.506210 13384 debug-util.cc:398] Leaking SignalData structure 0x7b080008df20 after lost signal to thread 12824
W20260202 15:08:31.511947 14521 log.cc:927] Time spent T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
W20260202 15:08:31.530089 12646 consensus_meta.cc:287] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.514s user 0.005s sys 0.005s
I20260202 15:08:31.511417 14287 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260202 15:08:31.557754 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 63651us
I20260202 15:08:31.570219 12690 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.538743 14451 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:31.541821 14459 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:31.573964 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 83915.4us
W20260202 15:08:31.567781 14530 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.216s user 0.001s sys 0.000s
W20260202 15:08:31.593562 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 102848us
I20260202 15:08:31.551002 12646 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:31.599481 13328 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.545030 14482 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:31.600044 14451 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:31.603981 14455 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:31.605067 14484 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:31.607494 12690 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:31.507906 14450 consensus_queue.cc:237] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:31.600368 13384 debug-util.cc:398] Leaking SignalData structure 0x7b08003e0ea0 after lost signal to thread 13387
I20260202 15:08:31.611541 12645 raft_consensus.cc:1275] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260202 15:08:31.601022 14529 log.cc:927] Time spent T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.243s user 0.000s sys 0.001s
W20260202 15:08:31.617671 14540 log.cc:927] Time spent T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
W20260202 15:08:31.618901 14542 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
W20260202 15:08:31.619390 14520 log.cc:927] Time spent T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260202 15:08:31.619266 14505 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.620196 14549 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:31.621986 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:31.622653 13362 raft_consensus.cc:2393] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 2: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:31.613765 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:31.595714 14287 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:31.625080 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:31.626623 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:31.628531 12795 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:31.628916 13328 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.602344 14451 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.630777 12690 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.633554 12690 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.634361 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:31.635723 12792 raft_consensus.cc:1275] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260202 15:08:31.637953 12722 raft_consensus.cc:1240] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:31.643391 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:31.667237 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:31.668133 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:31.684408 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:31.684497 12648 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:31.642998 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:31.645681 12690 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.636394 14354 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:31.636399 14158 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:31.627408 14533 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:31.668503 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:31.637287 14509 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:31.686251 12722 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:31.689380 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:31.601714 14411 consensus_queue.cc:237] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:31.702478 12798 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:31.631125 14505 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:31.627264 14475 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:31.708246 12795 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
W20260202 15:08:31.691563 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 51495.8us
W20260202 15:08:31.691843 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.692358 14268 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:31.667649 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:31.610936 14484 consensus_queue.cc:1059] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260202 15:08:31.696816 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:31.695588 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 66725.4us
I20260202 15:08:31.687834 12726 raft_consensus.cc:1275] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260202 15:08:31.701470 12796 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260202 15:08:31.644258 14481 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:31.713531 12722 raft_consensus.cc:1275] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260202 15:08:31.713609 12721 raft_consensus.cc:1240] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:31.714025 12800 raft_consensus.cc:1240] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:31.713939 14475 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.700423 12643 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:31.714567 14268 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:31.716868 12717 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20260202 15:08:31.717561 12648 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:31.713639 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:31.719455 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.723730 12798 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:31.729001 13328 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:31.730787 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 91625.2us
I20260202 15:08:31.699532 14458 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:31.780233 12718 raft_consensus.cc:2393] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:31.734977 12615 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:31.777673 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:31.781807 13361 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:31.700340 14500 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:31.700369 12646 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:31.721477 13358 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:31.812114 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:31.812642 12726 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:31.812927 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 184764us
I20260202 15:08:31.796464 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:31.813534 13326 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:31.796792 13361 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:31.823657 12689 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:31.738240 12797 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:31.698549 14343 consensus_queue.cc:1048] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260202 15:08:31.755633 12717 pending_rounds.cc:85] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Aborting all ops after (but not including) 3
W20260202 15:08:31.811463 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.777963 14268 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:31.789175 12690 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.805714 12793 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:31.832204 12794 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:31.812114 14498 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:31.703473 14533 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:31.839257 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 50189us
W20260202 15:08:31.842193 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 107342us
W20260202 15:08:31.842849 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.849068 12690 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:31.856277 14500 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:31.819561 14559 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:31.856630 14500 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:31.861967 12690 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:31.863240 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 134304us
W20260202 15:08:31.861824 14550 log.cc:927] Time spent T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.162s user 0.000s sys 0.000s
I20260202 15:08:31.838631 14514 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:31.870419 14451 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:31.856684 14316 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:31.870924 14498 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:31.837105 12717 pending_rounds.cc:107] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20260202 15:08:31.825894 14484 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:31.877816 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:31.881423 12723 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:31.901384 12798 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:31.899786 12688 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:31.937404 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:31.937732 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:31.939229 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 77322.9us
W20260202 15:08:31.952334 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 52626.7us
W20260202 15:08:31.955387 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:31.899600 14500 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
W20260202 15:08:31.936755 14503 consensus_meta.cc:287] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.659s user 0.011s sys 0.000s
I20260202 15:08:31.908416 14363 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:32.004967 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:31.936317 14158 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:31.909013 14316 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:31.907902 14482 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:32.006959 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 51795.6us
I20260202 15:08:31.953922 14514 raft_consensus.cc:697] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:31.992523 14373 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:31.957753 14563 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:31.994051 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:31.958467 14287 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:31.990427 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:31.993189 12717 raft_consensus.cc:2917] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20260202 15:08:32.009846 14343 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.026418 14459 raft_consensus.cc:2749] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:32.059379 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 245891us
I20260202 15:08:31.990659 14451 consensus_queue.cc:1059] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260202 15:08:32.037992 14316 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:31.993552 14484 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:31.899169 14559 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.014879 14259 consensus_queue.cc:1059] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260202 15:08:32.047313 14503 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.078287 13328 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:32.069514 14529 log.cc:927] Time spent T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260202 15:08:32.047600 14563 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:32.082811 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 232750us
I20260202 15:08:32.069514 12721 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:32.053613 14475 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:32.052316 14287 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:32.069592 14363 raft_consensus.cc:697] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:32.076486 12717 raft_consensus.cc:2889] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:32.122579 13360 raft_consensus.cc:1240] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:32.125622 14565 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
W20260202 15:08:32.126266 14560 log.cc:927] Time spent T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260202 15:08:32.126219 14565 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.131814 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:32.132529 13363 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:32.048763 14373 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.135939 12726 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
W20260202 15:08:32.136730 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 48950.3us
I20260202 15:08:32.120044 14458 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:32.138216 14450 consensus_queue.cc:1059] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:08:32.139264 14450 raft_consensus.cc:3055] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:32.139547 14450 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:32.140216 14450 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:32.143568 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 65142.2us
I20260202 15:08:32.145136 13328 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:32.193775 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 51453.4us
W20260202 15:08:32.199972 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 207316us
I20260202 15:08:32.200688 12690 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:32.203621 12690 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:32.088769 14563 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:32.123051 14287 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:32.076123 14343 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:32.220351 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 75309.3us
I20260202 15:08:32.221007 13328 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:32.252291 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:32.271297 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 60484us
W20260202 15:08:32.272980 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 52039.4us
I20260202 15:08:32.273784 13328 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:32.086032 14503 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:32.282805 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 71645us
I20260202 15:08:32.211035 14190 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.270893 14482 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:32.222358 12793 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:32.280903 14459 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:32.211002 14535 consensus_queue.cc:1059] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260202 15:08:32.299119 14450 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:32.299010 14258 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260202 15:08:32.310271 14533 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:32.309320 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1322) took 1202 ms. Trace:
W20260202 15:08:32.371150 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 49254us
W20260202 15:08:32.373477 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 169819us
I20260202 15:08:32.374675 12690 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:32.374899 12615 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:32.221805 14514 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.382287 12615 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:32.371855 14484 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:32.306619 14190 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.364037 14459 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:32.364085 14567 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:32.312966 14341 consensus_queue.cc:1059] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260202 15:08:32.417398 14567 raft_consensus.cc:697] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:08:32.418051 14573 log.cc:927] Time spent T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
W20260202 15:08:32.419174 14572 log.cc:927] Time spent T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260202 15:08:32.417423 14581 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:32.397709 14579 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:32.335991 14451 consensus_queue.cc:1048] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260202 15:08:32.394284 14533 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:32.405908 14373 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:32.432482 14343 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.363495 14458 consensus_queue.cc:1048] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
W20260202 15:08:32.433274 14562 log.cc:927] Time spent T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.427s user 0.000s sys 0.001s
I20260202 15:08:32.423166 14455 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:32.445973 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:32.329984 14363 consensus_queue.cc:237] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.422976 14484 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:32.503446 14565 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.219856 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:32.508383 14548 raft_consensus.cc:3055] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:32.508723 14548 raft_consensus.cc:740] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:32.509377 14548 consensus_queue.cc:260] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:32.510082 12725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.006s sys 0.009s
W20260202 15:08:32.510334 12725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.979s user 0.006s sys 0.009s
W20260202 15:08:32.512735 14268 consensus_meta.cc:287] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.640s user 0.006s sys 0.000s
I20260202 15:08:32.513082 14268 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:32.514145 14492 consensus_meta.cc:287] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.138s user 0.006s sys 0.000s
W20260202 15:08:32.516621 14582 log.cc:927] Time spent T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20260202 15:08:32.517164 14552 log.cc:927] Time spent T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.817s user 0.003s sys 0.005s
I20260202 15:08:32.517575 14583 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.518041 14583 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.519155 12720 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:32.519855 14583 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.510439 12721 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
W20260202 15:08:32.520620 13358 consensus_meta.cc:287] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.702s user 0.004s sys 0.005s
I20260202 15:08:32.521016 14583 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:32.521169 13358 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20260202 15:08:32.521723 13358 pending_rounds.cc:85] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Aborting all ops after (but not including) 3
I20260202 15:08:32.521955 13358 pending_rounds.cc:107] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
W20260202 15:08:32.522256 14580 log.cc:927] Time spent T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260202 15:08:32.522295 13358 raft_consensus.cc:2917] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20260202 15:08:32.522894 13358 raft_consensus.cc:2889] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:32.523187 14509 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:32.523312 14571 consensus_queue.cc:1048] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260202 15:08:32.523648 14509 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.525643 14509 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:32.526124 14585 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.526538 14585 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.528303 14585 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:32.529039 14474 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260202 15:08:32.535023 14573 log.cc:927] Time spent T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260202 15:08:32.537250 14190 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.544749 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1328) took 1048 ms. Trace:
I20260202 15:08:32.547724 14455 raft_consensus.cc:697] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:32.548733 14455 consensus_queue.cc:237] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.549291 14451 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.549813 14451 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.454092 14498 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.454558 14343 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.551519 14451 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:32.551388 14343 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.553298 14343 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:32.518332 14500 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.372840 12717 rpcz_store.cc:276] 0202 15:08:31.102772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.117762 (+ 14990us) service_pool.cc:225] Handling call
0202 15:08:31.142269 (+ 24507us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:31.499454 (+357185us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b5400184d20. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4dd09f 00007f24ed555be0 00007f24ed55745e 00007f24ed5617c9 00007f24ed562096 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2
0202 15:08:32.278152 (+778698us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:32.304947 (+ 26795us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9748864}
I20260202 15:08:32.556632 14570 raft_consensus.cc:3055] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:32.557039 14570 raft_consensus.cc:740] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:32.557844 14570 consensus_queue.cc:260] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.560384 14316 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.559844 14500 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:32.545246 12725 rpcz_store.cc:276] 0202 15:08:31.496054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.496159 (+ 105us) service_pool.cc:225] Handling call
0202 15:08:31.530510 (+ 34351us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:31.530855 (+ 345us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:31.530882 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:31.531316 (+ 434us) log.cc:844] Serialized 26 byte log entry
0202 15:08:32.510623 (+979307us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:32.510658 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:32.510712 (+ 54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:32.544542 (+ 33830us) raft_consensus.cc:1685] finished
0202 15:08:32.544560 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:32.544690 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":430,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":7127,"thread_start_us":979022,"threads_started":1,"wal-append.queue_time_us":480926}
I20260202 15:08:32.572192 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:32.441632 14567 consensus_queue.cc:237] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.572839 13358 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:32.450531 14579 raft_consensus.cc:697] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:32.510517 14548 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:32.581683 12689 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:32.521423 14583 raft_consensus.cc:697] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:32.581614 14268 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:32.577126 14503 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:32.652480 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:32.598850 14570 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:32.654876 14598 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.643862 14500 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260202 15:08:32.695629 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:32.702661 13326 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.023s sys 0.046s
W20260202 15:08:32.704721 13326 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.024s sys 0.046s
W20260202 15:08:32.707878 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 310200us
I20260202 15:08:32.690673 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:32.693131 14500 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:32.729759 12615 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:32.736677 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:32.740482 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:32.744051 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:32.747375 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 372730us
I20260202 15:08:32.748713 12690 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:32.762998 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:32.753577 12614 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:32.765872 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:32.731249 14500 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:32.712844 14559 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:32.770555 12649 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:32.769618 12615 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:32.746887 14450 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:32.773808 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:32.774933 12615 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:32.771492 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1332) took 1149 ms. Trace:
I20260202 15:08:32.764070 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:32.792153 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:32.779444 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:32.774976 14341 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.801433 12690 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:32.763818 14451 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260202 15:08:32.811686 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:32.778903 12719 rpcz_store.cc:276] 0202 15:08:31.621564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.627170 (+ 5606us) service_pool.cc:225] Handling call
0202 15:08:32.690530 (+1063360us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b58001d7098. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:32.770535 (+ 80005us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6498048}
I20260202 15:08:32.800163 14503 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.813464 14341 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.809298 14535 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:32.814234 14503 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.814769 14535 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:32.815467 14578 log.cc:927] Time spent T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.733s user 0.003s sys 0.000s
I20260202 15:08:32.815653 14341 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:32.816339 14503 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:32.816661 14535 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:32.817111 14495 raft_consensus.cc:2804] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:32.817466 14495 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:32.817683 14341 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:32.817752 14495 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:32.818334 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1720) took 1115 ms. Trace:
I20260202 15:08:32.779166 14306 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election won for term 4
W20260202 15:08:32.780031 14554 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.784s user 0.000s sys 0.002s
I20260202 15:08:32.775926 14373 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:32.702611 14598 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.828022 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:32.828027 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:32.831305 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:32.779959 14592 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:32.831867 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:32.842475 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:32.831226 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:32.845357 14555 raft_consensus.cc:2804] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:32.845459 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:32.845963 12798 raft_consensus.cc:1240] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:32.846751 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:32.846947 14594 log.cc:927] Time spent T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20260202 15:08:32.847214 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:32.847766 12798 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:32.847800 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:32.848073 14591 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.848950 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:32.852173 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:32.853008 12794 raft_consensus.cc:2393] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 2: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:32.845923 14555 raft_consensus.cc:697] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
W20260202 15:08:32.826732 14563 consensus_meta.cc:287] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.548s user 0.007s sys 0.003s
I20260202 15:08:32.848608 14591 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.806648 14450 raft_consensus.cc:697] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:32.828742 14585 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.864871 14585 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.813009 14316 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.866691 14585 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.829941 14373 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:32.870424 12764 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:32.863415 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:32.802984 14287 consensus_meta.cc:287] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.522s user 0.006s sys 0.003s
I20260202 15:08:32.875561 12800 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:08:32.831243 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:32.871104 14585 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.884167 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:32.809716 14484 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260202 15:08:32.871587 12796 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:32.814069 14559 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260202 15:08:32.877583 12764 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:32.880369 13328 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.000s sys 0.038s
W20260202 15:08:32.892100 13328 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s user 0.000s sys 0.038s
I20260202 15:08:32.827960 14343 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:32.892652 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 618883us
I20260202 15:08:32.836433 14306 raft_consensus.cc:697] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:32.883101 12765 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:32.900243 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1408) took 1523 ms. Trace:
I20260202 15:08:32.900671 12642 rpcz_store.cc:276] 0202 15:08:31.376758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.376983 (+ 225us) service_pool.cc:225] Handling call
0202 15:08:31.395887 (+ 18904us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:31.401950 (+ 6063us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:31.401982 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:31.435036 (+ 33054us) log.cc:844] Serialized 26 byte log entry
0202 15:08:31.570176 (+135140us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:31.570227 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:31.570308 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:32.582569 (+1012261us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:32.899971 (+317402us) raft_consensus.cc:1685] finished
0202 15:08:32.899997 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:32.900183 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5676,"raft.queue_time_us":43421,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":44006,"spinlock_wait_cycles":14418304,"thread_start_us":27942,"threads_started":1,"wal-append.queue_time_us":185475}
I20260202 15:08:32.884871 12721 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:32.718068 14579 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:32.859418 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:32.765211 14583 consensus_queue.cc:237] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.893716 13328 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:32.861105 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:32.874096 14373 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:32.877560 14592 raft_consensus.cc:697] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:32.886449 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:32.879858 14150 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.910607 12647 raft_consensus.cc:2368] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3.
I20260202 15:08:32.851802 14505 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260202 15:08:32.880692 14510 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.909761 14599 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260202 15:08:32.929478 13362 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:32.899003 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:32.830437 14341 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.920923 14591 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.921916 14598 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:32.922004 14606 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:32.922406 14555 consensus_queue.cc:237] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.921759 13328 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:32.947893 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:32.948470 12642 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:32.948820 13328 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:32.926013 12796 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:32.883882 14287 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.927199 14306 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.886757 14514 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:32.919921 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:32.896212 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:32.928133 14601 raft_consensus.cc:2749] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:32.952014 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:32.952457 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:32.891113 12797 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:32.919963 14581 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:32.947621 14510 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:32.973352 13355 raft_consensus.cc:1240] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:32.974116 14510 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:32.947518 14150 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.976325 14150 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.977264 12723 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:32.890085 14585 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.972055 14603 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:32.972110 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:32.972620 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:32.985700 13362 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:32.869498 14563 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.969794 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:32.987627 12642 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:32.981501 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:32.905156 14343 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.003532 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.004154 12723 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:32.953083 14535 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:33.006523 12689 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:32.969453 14598 raft_consensus.cc:2749] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:33.010524 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.011296 12726 raft_consensus.cc:2393] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:33.013211 14549 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:33.004752 14484 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:33.013679 14549 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.014107 14592 consensus_queue.cc:237] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.015645 14549 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.015734 14354 raft_consensus.cc:3055] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:33.016269 14354 raft_consensus.cc:740] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:33.017130 14354 consensus_queue.cc:260] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.018910 14500 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.005010 14602 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:33.021046 14602 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.021595 14569 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:33.022812 14602 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.023381 14486 raft_consensus.cc:3055] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:33.023702 14486 raft_consensus.cc:740] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:33.024262 14585 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.008941 14599 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.024410 14486 consensus_queue.cc:260] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.025072 14599 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.027117 14451 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.027544 14451 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.029326 14451 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:32.967636 14487 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260202 15:08:33.036634 12795 raft_consensus.cc:2410] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:32.987398 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 480) took 1820 ms. Trace:
I20260202 15:08:33.042276 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1414) took 1039 ms. Trace:
I20260202 15:08:33.042626 12648 rpcz_store.cc:276] 0202 15:08:32.003301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.003484 (+ 183us) service_pool.cc:225] Handling call
0202 15:08:32.040002 (+ 36518us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:32.291028 (+251026us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:32.291065 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:32.291582 (+ 517us) log.cc:844] Serialized 26 byte log entry
0202 15:08:32.398940 (+107358us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:32.399017 (+ 77us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:33.031190 (+632173us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54001dfbd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:33.031229 (+ 39us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:33.031271 (+ 42us) raft_consensus.cc:1685] finished
0202 15:08:33.031285 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:33.042199 (+ 10914us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17155,"raft.queue_time_us":14787,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":5648,"spinlock_wait_cycles":418993024,"thread_start_us":106901,"threads_started":1,"wal-append.queue_time_us":354470}
I20260202 15:08:33.031908 13356 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:32.978792 14514 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:32.968698 14450 consensus_queue.cc:237] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:33.046411 14482 consensus_meta.cc:287] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.736s user 0.010s sys 0.000s
I20260202 15:08:33.052834 12688 leader_election.cc:400] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3.
I20260202 15:08:33.059158 12688 leader_election.cc:403] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260202 15:08:33.064581 12615 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:33.073189 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 190142us
I20260202 15:08:33.119716 12724 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:33.013747 14484 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:33.122370 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 227027us
W20260202 15:08:33.122830 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 70109.4us
W20260202 15:08:33.123127 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 58622.2us
I20260202 15:08:33.123534 12764 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:33.123803 12688 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.045224 14591 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:33.123895 12615 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:33.130280 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.132323 12723 raft_consensus.cc:2393] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 2: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:33.133589 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.041095 14474 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20260202 15:08:33.058429 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.141724 12765 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.057340 14268 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.057590 14258 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.124347 14600 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:33.119535 14482 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:33.168584 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:33.169305 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:33.122440 14569 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:33.157817 12765 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.125516 14350 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
W20260202 15:08:33.171020 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 266696us
I20260202 15:08:33.179850 13360 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:33.182781 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.137377 14559 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:33.149385 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.148696 14599 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.152215 14585 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260202 15:08:33.131888 14567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.001s sys 0.026s
I20260202 15:08:33.187325 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.161049 14598 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:33.165227 14549 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:33.197070 12725 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:08:33.247445 12796 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:33.248085 12648 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:33.122570 14565 consensus_queue.cc:1059] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
W20260202 15:08:33.248423 14567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.002s sys 0.029s
I20260202 15:08:33.255062 12798 raft_consensus.cc:1275] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:33.255287 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.185106 14350 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.257695 12726 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:32.818727 12791 rpcz_store.cc:276] 0202 15:08:31.702835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.718661 (+ 15826us) service_pool.cc:225] Handling call
0202 15:08:31.718937 (+ 276us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:31.738022 (+ 19085us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:31.738068 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:31.856640 (+118572us) log.cc:844] Serialized 26 byte log entry
0202 15:08:31.990341 (+133701us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:32.339610 (+349269us) log.cc:844] Serialized 14 byte log entry
0202 15:08:32.703972 (+364362us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:32.710945 (+ 6973us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:32.814843 (+103898us) spinlock_profiling.cc:243] Waited 4.84 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:32.818029 (+ 3186us) raft_consensus.cc:1685] finished
0202 15:08:32.818119 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:32.818262 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5610,"prepare.queue_time_us":283939,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":214714,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":42326,"spinlock_wait_cycles":11136384,"thread_start_us":630911,"threads_started":3,"wal-append.queue_time_us":175779}
I20260202 15:08:33.180192 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:33.140666 14591 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.260185 12765 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:33.179847 14500 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.179875 14602 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20260202 15:08:33.262451 14616 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.185597 14502 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:33.039423 13354 rpcz_store.cc:276] 0202 15:08:31.164247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.207620 (+ 43373us) service_pool.cc:225] Handling call
0202 15:08:31.530603 (+322983us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:31.547915 (+ 17312us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:31.627168 (+ 79253us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:31.631278 (+ 4110us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:31.667261 (+ 35983us) log.cc:844] Serialized 26 byte log entry
0202 15:08:31.956297 (+289036us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:32.045785 (+ 89488us) log.cc:844] Serialized 14 byte log entry
0202 15:08:32.684082 (+638297us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b18001e3a00. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4ae335 00007f24ed556ce0 00007f24ed569396 00007f24ed579f07 00007f24ed579e88
0202 15:08:32.909609 (+225527us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:32.949184 (+ 39575us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:32.951581 (+ 2397us) raft_consensus.cc:1685] finished
0202 15:08:32.951597 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:32.984403 (+ 32806us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38929,"prepare.queue_time_us":21456,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":1792,"raft.queue_time_us":143080,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":75607,"spinlock_wait_cycles":15689856,"thread_start_us":331983,"threads_started":2,"wal-append.queue_time_us":416318}
I20260202 15:08:33.169592 14268 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.273367 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.273521 14618 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:33.172894 14258 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.273840 14614 raft_consensus.cc:2749] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:33.254807 14482 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:33.309828 14617 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.310678 14613 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.311287 14613 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.252408 14341 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.217048 14287 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.313846 14620 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.274595 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:33.217406 14585 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:33.314399 14620 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.315101 14585 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:33.315289 12796 raft_consensus.cc:2393] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:33.316078 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:33.251076 12720 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:33.216704 14535 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.261662 14598 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.257452 14595 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:33.217619 14559 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.270005 14363 raft_consensus.cc:3055] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:33.329959 14591 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.330207 14363 raft_consensus.cc:740] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:33.268240 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:33.180759 14600 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.276250 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.276789 14343 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:33.252117 14549 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:33.310278 14617 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:33.255931 14563 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:33.188304 14569 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.287079 14579 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:33.324241 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:33.324693 12763 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:33.325991 14350 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.327021 14620 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.327458 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.328073 14613 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.317277 14621 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:33.328414 14514 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:33.329222 14495 consensus_meta.cc:287] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.511s user 0.008s sys 0.000s
I20260202 15:08:33.332005 14598 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.331459 12794 raft_consensus.cc:1275] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260202 15:08:33.322546 14595 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:33.416788 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 274) took 1274 ms. Trace:
I20260202 15:08:33.329895 14559 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.299364 12797 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:33.378181 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:33.287612 14616 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.368921 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:33.387279 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 62675us
I20260202 15:08:33.371974 13358 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
W20260202 15:08:33.423286 14623 log.cc:927] Time spent T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260202 15:08:33.331560 14535 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:33.425060 14625 log.cc:927] Time spent T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260202 15:08:33.382377 14549 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:33.415732 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:33.385512 14601 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.418457 12614 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:33.321854 14604 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.418942 12794 pending_rounds.cc:85] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2: Aborting all ops after (but not including) 1
W20260202 15:08:33.370100 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 218456us
I20260202 15:08:33.390193 14579 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.436668 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:33.382911 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:33.437877 14495 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.438863 14595 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:33.621457 13357 raft_consensus.cc:2393] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 2: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:33.381451 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:33.380573 14570 consensus_meta.cc:287] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.691s user 0.015s sys 0.000s
I20260202 15:08:33.419275 12646 rpcz_store.cc:276] 0202 15:08:32.142049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.142431 (+ 382us) service_pool.cc:225] Handling call
0202 15:08:32.142678 (+ 247us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:32.142954 (+ 276us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:08:32.218916 (+ 75962us) log.cc:844] Serialized 14 byte log entry
0202 15:08:33.273256 (+1054340us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b1800343a20. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea42646a 00007f24f3aac5c9 00007f24ea2e6160 00007f24ea4ae3c4 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe5da 00007f24ed5696aa
0202 15:08:33.390632 (+117376us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:33.416562 (+ 25930us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:33.416599 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:33.416655 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:33.416744 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52801,"prepare.queue_time_us":78983,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":24996,"spinlock_wait_cycles":6756608,"thread_start_us":116401,"threads_started":1}
I20260202 15:08:33.382666 14343 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.416821 12796 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:33.586115 13360 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:33.583065 14536 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:33.389295 14484 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:33.591393 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:33.595323 12794 pending_rounds.cc:107] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2: Aborting uncommitted NO_OP operation due to leader change: 3.2
W20260202 15:08:33.591519 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:33.608947 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 190578us
I20260202 15:08:33.590870 14591 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.605155 14613 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.610154 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:33.585091 12720 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:33.610441 14601 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.620256 14579 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.622103 12721 raft_consensus.cc:2368] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:33.589861 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 513) took 1513 ms. Trace:
W20260202 15:08:33.590369 14624 log.cc:927] Time spent T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.325s user 0.001s sys 0.008s
I20260202 15:08:33.635741 14616 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.659487 14598 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.681488 14605 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:33.681967 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.591073 14559 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.683295 14559 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.683883 14617 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.660099 14570 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.684471 14570 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.685839 14559 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.632885 14604 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.585705 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.660918 14495 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.603480 14468 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:33.687614 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.688690 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.690287 12720 raft_consensus.cc:2393] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:33.584376 14363 consensus_queue.cc:260] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.670406 12689 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:33.673043 12614 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:33.603142 14514 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.636518 14621 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:33.680792 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:33.677014 13356 rpcz_store.cc:276] 0202 15:08:32.076745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.076873 (+ 128us) service_pool.cc:225] Handling call
0202 15:08:32.085261 (+ 8388us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:33.051731 (+966470us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:33.589573 (+537842us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b54005ed154. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:33.589790 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2111444352}
I20260202 15:08:33.683718 12765 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:33.636869 12794 raft_consensus.cc:2889] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:33.637686 13354 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:33.681165 14632 log.cc:927] Time spent T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260202 15:08:33.715451 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:33.712074 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.686391 14570 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.712221 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:33.709779 14373 consensus_meta.cc:287] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.757s user 0.015s sys 0.001s
I20260202 15:08:33.714955 14495 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:33.714708 14258 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.691653 14579 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:33.662103 14343 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.712190 13357 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 5
I20260202 15:08:33.676451 14591 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.707994 14621 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:33.701586 14559 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.708869 14268 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.707280 14569 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.731007 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:33.666657 14536 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:33.728297 14495 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:33.676576 14613 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.754572 14601 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.728976 14363 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:33.759773 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.737510 14293 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:33.756390 13326 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.713047 14341 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
I20260202 15:08:33.763949 14591 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:33.765484 14548 consensus_meta.cc:287] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.178s user 0.011s sys 0.006s
W20260202 15:08:33.765741 14577 log.cc:927] Time spent T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.908s user 0.003s sys 0.000s
I20260202 15:08:33.750047 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.709785 14514 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.769047 14536 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:33.777952 13360 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:33.721803 14535 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:33.731935 14373 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.793296 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:33.769439 14495 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:33.780954 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.787895 12644 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:33.794492 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1624) took 2156 ms. Trace:
I20260202 15:08:33.794940 12718 rpcz_store.cc:276] 0202 15:08:31.637758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.637856 (+ 98us) service_pool.cc:225] Handling call
0202 15:08:33.794336 (+2156480us) spinlock_profiling.cc:243] Waited 71.3 ms on lock 0x7b54003282d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:33.794441 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":164051712}
I20260202 15:08:33.789837 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:33.775173 14293 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:33.788255 13326 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.793835 13363 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:33.776273 14569 raft_consensus.cc:2749] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:33.797199 13362 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:33.801661 13326 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.802537 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.807142 12614 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:33.808171 12614 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:33.809490 12688 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.765785 14559 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.814160 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1412) took 1921 ms. Trace:
I20260202 15:08:33.814428 13356 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:33.814949 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.789219 14591 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.789043 14563 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.805605 14293 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:33.823853 14591 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:33.785445 14570 consensus_queue.cc:1059] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260202 15:08:33.829082 14607 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:33.809194 14535 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:33.808295 14450 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260202 15:08:33.814488 12643 rpcz_store.cc:276] 0202 15:08:31.892728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.901067 (+ 8339us) service_pool.cc:225] Handling call
0202 15:08:33.802513 (+1901446us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea47412e 000000000035a3c9 00007f24f249e560 00007f24f2566dd4 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:33.811975 (+ 9462us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:33.813806 (+ 1831us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:33.813857 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:33.813905 (+ 48us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:33.813945 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:33.814017 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:33.814111 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80026496}
W20260202 15:08:33.835220 14585 consensus_meta.cc:287] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.520s user 0.010s sys 0.001s
I20260202 15:08:33.852831 14585 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.855192 14585 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.856216 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:33.852008 14633 log.cc:927] Time spent T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20260202 15:08:33.825213 14600 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.837123 14563 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.859720 14233 raft_consensus.cc:3055] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:33.860174 14233 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:33.861218 14559 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.861281 14233 consensus_queue.cc:260] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.862268 14373 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.863214 14343 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.868865 14613 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.870307 12717 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:33.871476 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:33.872177 12717 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:33.872748 14309 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.873329 14309 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:33.877070 12800 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:33.879081 12800 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
W20260202 15:08:33.880422 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 821034us
W20260202 15:08:33.884301 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:33.843951 14591 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:33.845250 14644 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:33.828739 14638 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:33.842094 14341 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:33.851087 14641 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260202 15:08:33.851468 14618 consensus_meta.cc:287] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.529s user 0.007s sys 0.004s
I20260202 15:08:33.851526 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:33.851802 14514 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.851977 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.892786 12690 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.789073 14635 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:33.890390 12647 raft_consensus.cc:2368] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3.
I20260202 15:08:33.894393 12690 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.892441 14618 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3.
I20260202 15:08:33.860128 14563 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:33.897246 14585 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.897809 13356 raft_consensus.cc:2410] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:33.896569 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:33.913442 14545 log.cc:927] Time spent T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 2.363s user 0.011s sys 0.016s
I20260202 15:08:33.913769 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:33.914391 12643 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:33.914489 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1410) took 2609 ms. Trace:
I20260202 15:08:33.915207 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 508) took 2292 ms. Trace:
I20260202 15:08:33.915601 13359 rpcz_store.cc:276] 0202 15:08:31.622584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.627516 (+ 4932us) service_pool.cc:225] Handling call
0202 15:08:33.914494 (+2286978us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:33.914648 (+ 154us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:33.914858 (+ 210us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:33.914883 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:33.914942 (+ 59us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:33.914978 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:33.915037 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:33.915154 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33702272}
I20260202 15:08:33.890859 14343 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.914904 12650 rpcz_store.cc:276] 0202 15:08:31.305199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.305504 (+ 305us) service_pool.cc:225] Handling call
0202 15:08:31.325735 (+ 20231us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:31.326080 (+ 345us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:31.326109 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:31.326457 (+ 348us) log.cc:844] Serialized 26 byte log entry
0202 15:08:31.374836 (+ 48379us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:31.374879 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:31.374938 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:33.891481 (+2516543us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:33.914225 (+ 22744us) raft_consensus.cc:1685] finished
0202 15:08:33.914246 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:33.914422 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":579,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":8687,"spinlock_wait_cycles":32275328,"thread_start_us":35237,"threads_started":1,"wal-append.queue_time_us":93333}
I20260202 15:08:33.910171 12688 leader_election.cc:400] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3.
I20260202 15:08:33.902769 14233 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:33.889904 14341 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:33.889658 14638 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:33.919323 12688 leader_election.cc:403] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260202 15:08:33.907629 14268 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:33.905099 14514 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:33.921552 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.891670 14613 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20260202 15:08:33.921909 12690 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.887581 14644 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:33.927731 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.917552 12791 raft_consensus.cc:2368] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:33.902539 14495 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.926473 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.893373 14600 consensus_queue.cc:1059] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260202 15:08:33.923344 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.928555 12723 raft_consensus.cc:2393] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:33.929975 12650 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:33.931383 12690 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:33.922437 14638 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:33.922968 14341 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:33.939149 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:33.926455 14591 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:33.922268 13354 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:33.941731 12723 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:33.946087 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.917323 14585 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.925416 14563 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:33.947788 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.932870 14634 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:33.911223 14635 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.929430 14644 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:33.922711 14343 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.962328 12689 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:33.945789 14617 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:33.965210 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.947470 14258 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:33.985119 12763 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:33.936185 14268 consensus_queue.cc:1059] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260202 15:08:33.974771 14618 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:33.979804 14621 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.982905 14476 consensus_queue.cc:1059] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:33.983161 14343 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:33.986276 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:33.984820 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 381869us
I20260202 15:08:33.985143 12649 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 1.
I20260202 15:08:33.988055 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.957587 14634 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:33.968753 14647 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:33.994980 12615 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:33.971050 12650 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:33.996825 12763 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:34.000456 12688 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:33.959957 14563 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:33.989529 12717 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:33.992506 14648 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:34.069342 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:34.060902 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:33.998114 14634 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:34.069475 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 247488us
I20260202 15:08:34.075435 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:33.994138 14618 raft_consensus.cc:697] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:34.004279 12615 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:34.082408 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 78223.2us
I20260202 15:08:34.085355 13326 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:34.086864 12765 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:34.050611 14602 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:34.073181 14652 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.153337 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:34.075687 14648 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:34.159900 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1397) took 1020 ms. Trace:
I20260202 15:08:34.161314 12798 rpcz_store.cc:276] 0202 15:08:33.139240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.139352 (+ 112us) service_pool.cc:225] Handling call
0202 15:08:33.168012 (+ 28660us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:33.257431 (+ 89419us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.159668 (+902237us) spinlock_profiling.cc:243] Waited 86.7 ms on lock 0x7b540036e554. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:34.159839 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199442816}
I20260202 15:08:34.163287 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:34.157773 14617 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.164875 14617 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:34.165714 14459 raft_consensus.cc:3055] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:34.166114 14459 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:34.166903 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1413) took 2541 ms. Trace:
I20260202 15:08:34.157620 14652 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:34.166798 14459 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.167474 14607 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:34.167817 14459 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:34.168506 14652 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:34.169512 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1636) took 1373 ms. Trace:
I20260202 15:08:34.169821 12725 rpcz_store.cc:276] 0202 15:08:32.796458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.797387 (+ 929us) service_pool.cc:225] Handling call
0202 15:08:32.819409 (+ 22022us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:33.252226 (+432817us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.169329 (+917103us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54003ac5d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:34.169462 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":867758976}
I20260202 15:08:34.171016 14618 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.171599 14617 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:34.172712 14617 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:34.173185 14617 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.164016 12650 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:34.105422 14534 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:34.167265 12645 rpcz_store.cc:276] 0202 15:08:31.625982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.626113 (+ 131us) service_pool.cc:225] Handling call
0202 15:08:32.817711 (+1191598us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:32.817877 (+ 166us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:32.980461 (+162584us) raft_consensus.cc:1534] Early marking committed up to index 4
0202 15:08:33.212564 (+232103us) log.cc:844] Serialized 14 byte log entry
0202 15:08:33.377189 (+164625us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:33.377236 (+ 47us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:33.377282 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.166727 (+789445us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b540024ed54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:34.166765 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:34.166856 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10720,"prepare.queue_time_us":324859,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":12798,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":33162,"spinlock_wait_cycles":427014144,"thread_start_us":381352,"threads_started":2,"wal-append.queue_time_us":209125}
I20260202 15:08:34.094435 14476 raft_consensus.cc:2749] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:34.090335 14571 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:34.107432 13357 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:34.185467 14635 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:34.185768 14617 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:34.174841 14495 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:34.186954 12643 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:34.191331 14571 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:34.273895 14571 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:34.243139 14534 raft_consensus.cc:697] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:34.274168 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:34.147559 14484 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.187175 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34778 (request call id 1421) took 1481 ms. Trace:
I20260202 15:08:34.151189 14602 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:34.188014 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:34.069224 14647 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.108433 14645 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.191593 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:34.196897 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:34.197026 14268 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:34.162855 14648 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:34.199450 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:34.163000 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:34.239493 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 277250us
I20260202 15:08:34.161722 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:34.244506 12725 raft_consensus.cc:1275] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:34.240818 14607 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:34.182227 14649 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.285943 12641 rpcz_store.cc:276] 0202 15:08:32.706224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.706335 (+ 111us) service_pool.cc:225] Handling call
0202 15:08:34.186773 (+1480438us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001decd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c90c0 00007f24ed567bba 00007f24f251858e 00007f24ebab7437 00007f24ebab73c2 00007f24ebab72f1 00007f24ebab726c 00007f24ebab6522 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:34.187100 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2413611264}
I20260202 15:08:34.261922 14585 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:34.291604 12798 raft_consensus.cc:3060] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:34.292007 13326 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:34.315021 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:34.315716 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:34.316882 13326 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:34.241973 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:34.311655 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:34.286091 14602 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:34.274803 12643 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
W20260202 15:08:34.323112 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 255) took 1386 ms (client timeout 1572 ms). Trace:
I20260202 15:08:34.325439 13362 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:34.326507 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:34.308440 14476 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:34.327833 12721 rpcz_store.cc:269] 0202 15:08:32.936434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.940496 (+ 4062us) service_pool.cc:225] Handling call
0202 15:08:34.322885 (+1382389us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b54003acad4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:34.323042 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1502248320}
I20260202 15:08:34.335721 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:34.316743 13354 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:34.341718 12798 raft_consensus.cc:2468] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:34.342222 14571 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:34.342653 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:34.324733 14563 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:34.327280 14570 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.364120 14570 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:34.303828 14484 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:34.365306 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:34.324297 14176 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:34.343014 12690 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:34.321157 14495 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.319454 14569 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:34.372114 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:34.372449 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:34.326237 14592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.205s user 0.010s sys 0.010s
I20260202 15:08:34.383045 13328 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:34.371510 14610 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260202 15:08:34.379961 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:34.381420 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:34.439796 12725 raft_consensus.cc:1275] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260202 15:08:34.442782 14505 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.368443 14566 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:34.318274 14607 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.369482 14601 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:34.370118 14585 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:34.329339 14652 consensus_queue.cc:1048] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260202 15:08:34.317942 14645 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:34.329525 14649 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.345978 14476 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.382308 14563 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:34.316131 14534 consensus_queue.cc:237] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:34.620373 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 55333us
I20260202 15:08:34.383275 14495 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.676090 14484 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:34.792490 14644 consensus_meta.cc:287] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.826s user 0.007s sys 0.003s
W20260202 15:08:34.772241 14639 log.cc:927] Time spent T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
W20260202 15:08:34.799470 14638 consensus_meta.cc:287] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.848s user 0.009s sys 0.000s
I20260202 15:08:34.799396 14644 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.529109 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1421) took 1758 ms. Trace:
I20260202 15:08:34.375635 14657 raft_consensus.cc:2804] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election won for term 5
W20260202 15:08:34.402268 12690 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.596s (SENT) [suppressed 6 similar messages]
I20260202 15:08:34.580652 12646 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260202 15:08:34.546909 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:34.801270 12642 rpcz_store.cc:276] 0202 15:08:32.771038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.988236 (+217198us) service_pool.cc:225] Handling call
0202 15:08:33.011874 (+ 23638us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:33.033316 (+ 21442us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:33.033353 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:33.033401 (+ 48us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:33.033436 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.528922 (+1495486us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b54002a9c54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:34.528961 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:34.529058 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48885632}
W20260202 15:08:34.717913 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 195410us
W20260202 15:08:34.802765 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:34.443653 14655 log.cc:927] Time spent T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260202 15:08:34.476976 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:34.447724 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:34.800628 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 417603us
W20260202 15:08:34.385177 14592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.324s user 0.010s sys 0.010s
I20260202 15:08:34.825680 12798 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:34.829003 12688 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:34.436604 14268 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:34.371946 14176 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:34.667500 14549 consensus_meta.cc:287] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.045s user 0.007s sys 0.002s
W20260202 15:08:34.528718 14662 log.cc:927] Time spent T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20260202 15:08:34.666122 14667 log.cc:927] Time spent T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260202 15:08:34.630887 14536 consensus_meta.cc:287] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.832s user 0.010s sys 0.000s
W20260202 15:08:34.662061 14293 consensus_meta.cc:287] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.830s user 0.009s sys 0.000s
I20260202 15:08:34.570669 14571 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:34.579588 14659 log.cc:927] Time spent T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260202 15:08:34.857736 12647 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:34.859572 12645 raft_consensus.cc:2410] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:34.856855 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 278) took 1164 ms. Trace:
W20260202 15:08:34.580245 14658 log.cc:927] Time spent T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.195s user 0.000s sys 0.000s
I20260202 15:08:34.578977 14666 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.855850 14604 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:34.796896 14233 consensus_meta.cc:287] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.874s user 0.007s sys 0.001s
W20260202 15:08:34.682245 14459 consensus_meta.cc:287] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.514s user 0.007s sys 0.003s
W20260202 15:08:34.682912 14651 log.cc:927] Time spent T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.369s user 0.000s sys 0.000s
I20260202 15:08:34.669384 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:34.528597 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 695) took 1960 ms. Trace:
W20260202 15:08:34.798365 14341 consensus_meta.cc:287] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.845s user 0.010s sys 0.006s
W20260202 15:08:34.903949 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 501670us
I20260202 15:08:34.904542 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:34.905340 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:34.573534 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:34.674307 14316 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:34.561749 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:34.681485 14647 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:34.529943 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1399) took 1390 ms. Trace:
W20260202 15:08:34.906615 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 76578.5us
I20260202 15:08:34.460219 14505 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:34.799958 14638 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.476553 14601 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.831624 14585 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.819381 14563 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.578825 14656 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.676s
I20260202 15:08:34.858933 14268 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:34.758149 14637 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:34.842736 14566 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:34.907785 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:35.482563 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 257) took 2209 ms. Trace:
I20260202 15:08:35.321147 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 532) took 1241 ms. Trace:
W20260202 15:08:35.626432 12798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1401) took 1683 ms (client timeout 1782 ms). Trace:
W20260202 15:08:35.665510 12798 rpcz_store.cc:269] 0202 15:08:33.943314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.854399 (+911085us) service_pool.cc:225] Handling call
0202 15:08:35.626385 (+771986us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:35.666678 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:35.670974 12798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1739) took 1508 ms (client timeout 1666 ms). Trace:
W20260202 15:08:35.671936 12798 rpcz_store.cc:269] 0202 15:08:34.162429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:35.666446 (+1504017us) service_pool.cc:225] Handling call
0202 15:08:35.670901 (+ 4455us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:35.665436 13361 rpcz_store.cc:276] 0202 15:08:34.079645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.079791 (+ 146us) service_pool.cc:225] Handling call
0202 15:08:34.147840 (+ 68049us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:34.148228 (+ 388us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:34.148265 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:34.162787 (+ 14522us) log.cc:844] Serialized 52 byte log entry
0202 15:08:34.309144 (+146357us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:34.798288 (+489144us) log.cc:844] Serialized 14 byte log entry
0202 15:08:34.798556 (+ 268us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.798622 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:35.320911 (+522289us) raft_consensus.cc:1685] finished
0202 15:08:35.320936 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:35.321084 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33462,"prepare.queue_time_us":589889,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54169,"raft.queue_time_us":6186,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":22459,"thread_start_us":601992,"threads_started":2,"wal-append.queue_time_us":203158}
I20260202 15:08:35.437652 14495 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:35.541286 14514 consensus_meta.cc:287] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.613s user 0.010s sys 0.000s
I20260202 15:08:35.377560 14565 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:35.544436 14569 consensus_meta.cc:287] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.162s user 0.009s sys 0.000s
W20260202 15:08:35.486275 14595 consensus_meta.cc:287] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.865s user 0.012s sys 0.000s
W20260202 15:08:35.536828 13354 consensus_meta.cc:287] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.164s user 0.008s sys 0.000s
I20260202 15:08:35.628449 12797 rpcz_store.cc:276] 0202 15:08:33.139866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.421633 (+281767us) service_pool.cc:225] Handling call
0202 15:08:33.585975 (+164342us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:33.692627 (+106652us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:33.701985 (+ 9358us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:33.876760 (+174775us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x7b2c0003d748. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ebe5d9b1 00007f24ebe6cfe2 00007f24ed59a851 00007f24ed55f0ba 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22
0202 15:08:33.877120 (+ 360us) log.cc:844] Serialized 26 byte log entry
0202 15:08:34.079900 (+202780us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:34.318452 (+238552us) log.cc:844] Serialized 14 byte log entry
0202 15:08:34.357475 (+ 39023us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.371389 (+ 13914us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:34.378727 (+ 7338us) raft_consensus.cc:1685] finished
0202 15:08:34.454460 (+ 75733us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:34.515943 (+ 61483us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47492,"prepare.queue_time_us":223148,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":6110,"raft.queue_time_us":37143,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"spinlock_wait_cycles":28927744,"thread_start_us":322192,"threads_started":2,"wal-append.queue_time_us":287188}
W20260202 15:08:35.105546 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:34.907155 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:35.545214 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1729) took 2709 ms. Trace:
I20260202 15:08:35.696168 12794 rpcz_store.cc:276] 0202 15:08:32.835607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.867444 (+ 31837us) service_pool.cc:225] Handling call
0202 15:08:33.331101 (+463657us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:33.715885 (+384784us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:35.544931 (+1829046us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b54003723d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:35.545115 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1030565120}
I20260202 15:08:35.696753 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:35.698335 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:35.699577 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:35.700243 12642 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
W20260202 15:08:35.540320 14648 consensus_meta.cc:287] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.225s user 0.008s sys 0.003s
I20260202 15:08:35.543298 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:35.756376 12799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1395) took 2925 ms (client timeout 1596 ms). Trace:
W20260202 15:08:35.756718 12799 rpcz_store.cc:269] 0202 15:08:32.830719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.830878 (+ 159us) service_pool.cc:225] Handling call
0202 15:08:35.543149 (+2712271us) spinlock_profiling.cc:243] Waited 907 us on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:35.756297 (+213148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2085120}
I20260202 15:08:35.428470 14476 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:34.906852 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 77914.5us
I20260202 15:08:35.684597 14514 raft_consensus.cc:2749] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:34.944358 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:35.542829 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:35.627072 14354 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:35.540766 12546 catalog_manager.cc:5654] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 2 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:35.761258 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:35.760919 12546 catalog_manager.cc:5654] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 2 to 3, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 3 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:35.316593 14309 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:35.631357 12646 rpcz_store.cc:276] 0202 15:08:33.692193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.692318 (+ 125us) service_pool.cc:225] Handling call
0202 15:08:33.692627 (+ 309us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:34.580419 (+887792us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b5400309e20. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4d6df0 00007f24ed560e40 00007f24ed561eb1 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c
0202 15:08:34.856643 (+276224us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.856811 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34960640}
I20260202 15:08:35.630683 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 713) took 1297 ms. Trace:
I20260202 15:08:35.772615 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 267) took 2063 ms. Trace:
I20260202 15:08:35.772979 12722 rpcz_store.cc:276] 0202 15:08:33.709639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.754474 (+ 44835us) service_pool.cc:225] Handling call
0202 15:08:35.542638 (+1788164us) spinlock_profiling.cc:243] Waited 7.54 ms on lock 0x7b58001d7098. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:35.772563 (+229925us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17331456}
I20260202 15:08:35.639578 14666 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:35.050870 14661 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:34.522504 14673 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:35.314671 14503 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:35.486392 14602 consensus_meta.cc:287] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.148s user 0.005s sys 0.005s
I20260202 15:08:35.641426 14268 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:35.513813 14611 log.cc:927] Time spent T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 2.540s user 0.004s sys 0.009s
I20260202 15:08:35.641813 14505 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:34.905006 14293 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:35.627779 14644 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:35.643579 14316 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:35.019980 14681 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:35.627059 14607 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:35.311401 14688 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:35.646523 14566 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:35.771677 14476 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:35.647086 14638 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:34.855036 14559 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:35.648003 14585 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:35.648499 14341 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:35.124564 14591 consensus_meta.cc:287] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.174s user 0.008s sys 0.002s
I20260202 15:08:35.650018 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 281) took 1948 ms. Trace:
I20260202 15:08:35.650578 14637 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:35.643396 12615 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:35.651185 13355 rpcz_store.cc:276] 0202 15:08:32.568628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.571855 (+ 3227us) service_pool.cc:225] Handling call
0202 15:08:32.572136 (+ 281us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:32.973755 (+401619us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.528388 (+1554633us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b540045e2d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:34.528541 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":922141056}
I20260202 15:08:35.654162 14563 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:35.656258 14601 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:35.008275 14678 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:35.131547 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 259) took 1807 ms. Trace:
I20260202 15:08:35.137264 14652 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:35.658555 12764 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.008s sys 0.003s
I20260202 15:08:35.659088 14459 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:34.854974 14176 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:35.134639 14634 consensus_meta.cc:287] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.042s user 0.007s sys 0.002s
W20260202 15:08:35.660269 12793 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1400) took 1717 ms (client timeout 1760 ms). Trace:
I20260202 15:08:34.960067 14657 raft_consensus.cc:697] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:35.625571 12796 rpcz_store.cc:276] 0202 15:08:33.272753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.315814 (+ 43061us) service_pool.cc:225] Handling call
0202 15:08:35.482374 (+2166560us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b540004c254. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:35.482494 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230308608}
I20260202 15:08:35.538475 12547 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45944 (request call id 309) took 5569 ms. Trace:
W20260202 15:08:35.663964 12763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.019s sys 0.023s
W20260202 15:08:35.664256 12614 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.013s sys 0.024s
I20260202 15:08:35.664163 14617 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:35.664685 14158 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:35.666531 14701 raft_consensus.cc:670] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:35.447077 14645 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:35.130810 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 534) took 1043 ms. Trace:
I20260202 15:08:35.101624 14233 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:34.898494 14549 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:34.904512 14536 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:35.678400 13357 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:35.686059 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:35.689366 12765 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.001s sys 0.036s
I20260202 15:08:35.688979 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:35.701756 14569 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260202 15:08:35.701968 14583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.727s user 0.003s sys 0.018s
I20260202 15:08:35.763398 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:35.766260 12546 catalog_manager.cc:5654] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 3, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 3 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260202 15:08:35.774847 12766 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.808s user 0.018s sys 0.021s
I20260202 15:08:35.776644 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:35.701612 13354 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:35.774456 14656 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:35.784198 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1649) took 1448 ms. Trace:
I20260202 15:08:35.695039 14565 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:35.771628 13358 rpcz_store.cc:276] 0202 15:08:34.333230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.571029 (+237799us) service_pool.cc:225] Handling call
0202 15:08:35.630612 (+1059583us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:35.811338 14613 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.878s user 0.013s sys 0.026s
I20260202 15:08:35.777629 14514 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:35.756127 14595 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:35.760260 14648 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:35.818972 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:35.815383 14661 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:35.823256 14503 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:35.838351 12688 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:36.149976 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 130988us
W20260202 15:08:36.194223 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 550837us
I20260202 15:08:35.812130 12717 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:36.155066 13361 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:36.202664 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 716) took 1300 ms. Trace:
I20260202 15:08:36.203011 13361 rpcz_store.cc:276] 0202 15:08:34.902397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:35.678245 (+775848us) service_pool.cc:225] Handling call
0202 15:08:36.202612 (+524367us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:36.153074 12763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.198s user 0.030s sys 0.030s
I20260202 15:08:35.836410 14673 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
W20260202 15:08:36.155532 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 175616us
W20260202 15:08:36.156978 12766 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.196s user 0.026s sys 0.033s
W20260202 15:08:36.152657 12794 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1406) took 2205 ms (client timeout 1893 ms). Trace:
W20260202 15:08:36.208855 12794 rpcz_store.cc:269] 0202 15:08:33.947380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:35.697360 (+1749980us) service_pool.cc:225] Handling call
0202 15:08:36.152571 (+455211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.153760 12722 rpcz_store.cc:276] 0202 15:08:34.335413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:35.783372 (+1447959us) service_pool.cc:225] Handling call
0202 15:08:35.783697 (+ 325us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:35.783898 (+ 201us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:35.783918 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:35.783963 (+ 45us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:35.783985 (+ 22us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:35.784042 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:35.784145 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:36.156600 12765 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.251s user 0.004s sys 0.058s
I20260202 15:08:35.878834 12725 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
W20260202 15:08:35.848662 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:36.157279 12764 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.257s user 0.015s sys 0.003s
I20260202 15:08:36.156906 12799 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:08:36.210923 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 1.40841e+06us
W20260202 15:08:36.211117 12799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1410) took 2255 ms (client timeout 1937 ms). Trace:
I20260202 15:08:36.151793 12800 rpcz_store.cc:276] 0202 15:08:33.324084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.879804 (+555720us) service_pool.cc:225] Handling call
0202 15:08:33.880111 (+ 307us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:33.880459 (+ 348us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:33.880495 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:33.880976 (+ 481us) log.cc:844] Serialized 26 byte log entry
0202 15:08:34.242908 (+361932us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:34.320401 (+ 77493us) log.cc:844] Serialized 14 byte log entry
0202 15:08:34.322423 (+ 2022us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.322483 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:35.131320 (+808837us) raft_consensus.cc:1685] finished
0202 15:08:35.131347 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:35.131484 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1819,"prepare.queue_time_us":122250,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":3951,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":16859,"thread_start_us":377464,"threads_started":2,"wal-append.queue_time_us":435339,"wal-append.run_cpu_time_us":19673,"wal-append.run_wall_time_us":1831734}
W20260202 15:08:36.180423 12796 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 267) took 2217 ms (client timeout 1966 ms). Trace:
I20260202 15:08:36.156913 12648 rpcz_store.cc:276] 0202 15:08:33.701670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.701803 (+ 133us) service_pool.cc:225] Handling call
0202 15:08:33.715305 (+ 13502us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:33.718418 (+ 3113us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:33.718442 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:33.718863 (+ 421us) log.cc:844] Serialized 26 byte log entry
0202 15:08:33.964473 (+245610us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:33.964510 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:33.964567 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:35.649845 (+1685278us) raft_consensus.cc:1685] finished
0202 15:08:35.649865 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:35.649974 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":54768,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":3931,"thread_start_us":245299,"threads_started":1,"wal-append.queue_time_us":374185}
W20260202 15:08:36.211752 12796 rpcz_store.cc:269] 0202 15:08:33.955130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.170135 (+2215005us) service_pool.cc:212] Skipping call since client already timed out
0202 15:08:36.172326 (+ 2191us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260202 15:08:36.156658 14613 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.275s user 0.013s sys 0.027s
I20260202 15:08:35.859303 14607 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:36.183950 12641 raft_consensus.cc:1240] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:36.183272 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.219215 12795 raft_consensus.cc:2468] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 1.
W20260202 15:08:36.219805 12795 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 260) took 2895 ms (client timeout 1995 ms). Trace:
I20260202 15:08:35.928241 14644 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:36.170634 14706 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:36.163295 13357 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 714) took 1827 ms (client timeout 1664 ms). Trace:
W20260202 15:08:36.156119 14583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.182s user 0.010s sys 0.031s
I20260202 15:08:36.152132 14689 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:36.154685 14678 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:35.826663 14268 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:36.211598 12799 rpcz_store.cc:269] 0202 15:08:33.955325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:35.760464 (+1805139us) service_pool.cc:225] Handling call
0202 15:08:36.211043 (+450579us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.241343 14661 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:36.241505 14585 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
W20260202 15:08:36.241815 12799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1736) took 2094 ms (client timeout 2000 ms). Trace:
W20260202 15:08:36.242276 12799 rpcz_store.cc:269] 0202 15:08:34.147305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.241687 (+2094382us) service_pool.cc:212] Skipping call since client already timed out
0202 15:08:36.241769 (+ 82us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260202 15:08:36.242071 14585 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.243039 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.243458 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:36.173352 14158 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.244112 14585 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:35.847473 14681 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:36.244942 12764 connection.cc:232] RPC call timeout handler was delayed by -1.0520484989997385s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
W20260202 15:08:36.221361 12800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 269) took 1885 ms (client timeout 1795 ms). Trace:
I20260202 15:08:36.205909 14638 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:35.818759 14602 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:36.220219 12795 rpcz_store.cc:269] 0202 15:08:33.324428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.330603 (+ 6175us) service_pool.cc:225] Handling call
0202 15:08:36.183135 (+2852532us) spinlock_profiling.cc:243] Waited 3.7 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:36.219701 (+ 36566us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8504576}
I20260202 15:08:36.257016 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.257620 12795 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
W20260202 15:08:36.258311 12795 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1408) took 2306 ms (client timeout 2790 ms). Trace:
W20260202 15:08:36.258718 12795 rpcz_store.cc:269] 0202 15:08:33.951591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.256781 (+2305190us) service_pool.cc:225] Handling call
0202 15:08:36.258253 (+ 1472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.259601 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.260226 12795 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
W20260202 15:08:36.260696 12795 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1412) took 2017 ms (client timeout 2506 ms). Trace:
W20260202 15:08:36.261034 12795 rpcz_store.cc:269] 0202 15:08:34.243591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.259363 (+2015772us) service_pool.cc:225] Handling call
0202 15:08:36.260653 (+ 1290us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:36.222481 12794 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 268) took 2239 ms (client timeout 2004 ms). Trace:
I20260202 15:08:36.223068 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:36.160768 14535 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:35.864969 12643 raft_consensus.cc:1275] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260202 15:08:36.226970 14637 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:36.170329 14556 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
W20260202 15:08:36.180341 13354 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 533) took 2091 ms (client timeout 1526 ms). Trace:
I20260202 15:08:36.307884 14693 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:36.353232 13354 rpcz_store.cc:269] 0202 15:08:34.084490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.093345 (+ 8855us) service_pool.cc:225] Handling call
0202 15:08:36.174064 (+2080719us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:36.154392 12614 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.050s user 0.018s sys 0.040s
I20260202 15:08:36.153822 13359 rpcz_store.cc:276] 0202 15:08:34.087364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.090211 (+ 2847us) service_pool.cc:225] Handling call
0202 15:08:34.152226 (+ 62015us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:34.169426 (+ 17200us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:34.169456 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:34.169850 (+ 394us) log.cc:844] Serialized 26 byte log entry
0202 15:08:34.332893 (+163043us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:34.332940 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:34.333003 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:35.128097 (+795094us) raft_consensus.cc:1685] finished
0202 15:08:35.130585 (+ 2488us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:35.130748 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5516,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":7212,"thread_start_us":150690,"threads_started":1,"wal-append.queue_time_us":165727}
I20260202 15:08:36.177189 14634 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.184302 14563 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:36.237629 14601 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:36.240962 14678 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:36.243516 12799 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:36.168610 14591 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3.
I20260202 15:08:36.244611 14293 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:36.215131 12649 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 287) took 2474 ms (client timeout 1609 ms). Trace:
I20260202 15:08:36.221698 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:36.221144 12796 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1409) took 2257 ms (client timeout 2189 ms). Trace:
W20260202 15:08:36.369326 12796 rpcz_store.cc:269] 0202 15:08:33.955027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.212344 (+2257317us) service_pool.cc:212] Skipping call since client already timed out
0202 15:08:36.212437 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260202 15:08:36.245445 12800 rpcz_store.cc:269] 0202 15:08:34.328088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.211886 (+1883798us) service_pool.cc:212] Skipping call since client already timed out
0202 15:08:36.212014 (+ 128us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260202 15:08:36.241900 14661 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260202 15:08:36.248864 14158 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.249187 12719 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:36.239733 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:36.250084 14689 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.197101 14656 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.270172 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:36.262540 12794 rpcz_store.cc:269] 0202 15:08:33.969925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.209579 (+2239654us) service_pool.cc:212] Skipping call since client already timed out
0202 15:08:36.209702 (+ 123us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260202 15:08:36.327001 14695 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:36.377910 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1404) took 2433 ms. Trace:
I20260202 15:08:36.231458 14569 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:36.231729 14701 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:36.155360 12793 rpcz_store.cc:269] 0202 15:08:33.942927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.383911 (+440984us) service_pool.cc:225] Handling call
0202 15:08:35.660192 (+1276281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.179539 14514 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.160030 14559 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.172636 14476 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.173034 14459 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.329370 14708 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:36.181089 14233 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:36.251482 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:36.256456 14596 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:36.358024 12650 raft_consensus.cc:1240] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:36.175915 14617 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:08:36.362483 12799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 265) took 2432 ms (client timeout 2768 ms). Trace:
W20260202 15:08:36.353660 14609 log.cc:927] Time spent T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 3.380s user 0.010s sys 0.010s
I20260202 15:08:36.233094 14607 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.163300 14652 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:36.368528 12649 rpcz_store.cc:269] 0202 15:08:33.727905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.792287 (+ 64382us) service_pool.cc:225] Handling call
0202 15:08:36.201922 (+2409635us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b5400012cd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:36.202042 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":411117824}
I20260202 15:08:36.200704 14688 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.231739 14644 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.362169 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:36.241382 14268 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.378536 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:36.449923 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:36.378316 12795 rpcz_store.cc:276] 0202 15:08:33.944242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.261670 (+2317428us) service_pool.cc:225] Handling call
0202 15:08:36.377846 (+116176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.379559 13328 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:36.357056 14548 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:36.404829 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.405822 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:36.353932 14693 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.375710 14565 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.408399 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.158464 12547 rpcz_store.cc:276] 0202 15:08:29.969030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:29.972468 (+ 3438us) service_pool.cc:225] Handling call
0202 15:08:35.538401 (+5565933us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:08:30.676522 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:08:30.726005 (+ 49483us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:08:30.726069 (+ 64us) write_op.cc:435] Acquired schema lock
0202 15:08:30.726107 (+ 38us) tablet.cc:660] Decoding operations
0202 15:08:30.727355 (+ 1248us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:08:30.727452 (+ 97us) write_op.cc:641] Partition lock acquired for write op
0202 15:08:30.727492 (+ 40us) tablet.cc:683] Acquiring locks for 9 operations
0202 15:08:30.728448 (+ 956us) tablet.cc:699] Row locks acquired
0202 15:08:30.728484 (+ 36us) write_op.cc:260] PREPARE: finished
0202 15:08:30.728671 (+ 187us) write_op.cc:270] Start()
0202 15:08:30.728874 (+ 203us) write_op.cc:276] Timestamp: P: 1770044910728588 usec, L: 0
0202 15:08:30.728921 (+ 47us) op_driver.cc:348] REPLICATION: starting
0202 15:08:30.729657 (+ 736us) log.cc:844] Serialized 3674 byte log entry
0202 15:08:31.294921 (+565264us) op_driver.cc:464] REPLICATION: finished
0202 15:08:31.439573 (+144652us) write_op.cc:301] APPLY: starting
0202 15:08:31.495302 (+ 55729us) tablet.cc:1365] starting BulkCheckPresence
0202 15:08:31.572126 (+ 76824us) tablet.cc:1368] finished BulkCheckPresence
0202 15:08:31.607228 (+ 35102us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:08:32.015150 (+407922us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:08:32.128042 (+112892us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0202 15:08:32.128106 (+ 64us) write_op.cc:312] APPLY: finished
0202 15:08:32.128355 (+ 249us) log.cc:844] Serialized 71 byte log entry
0202 15:08:32.128516 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:32.128626 (+ 110us) write_op.cc:454] Released schema lock
0202 15:08:32.128677 (+ 51us) write_op.cc:341] FINISH: Updating metrics
0202 15:08:32.129037 (+ 360us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:32.129061 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12219,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":99438,"apply.run_cpu_time_us":13148,"apply.run_wall_time_us":697486,"num_ops":9,"prepare.queue_time_us":41913,"prepare.run_cpu_time_us":19852,"prepare.run_wall_time_us":280850,"raft.queue_time_us":153755,"raft.run_cpu_time_us":23236,"raft.run_wall_time_us":63691,"replication_time_us":565836,"thread_start_us":343377,"threads_started":4,"wal-append.queue_time_us":387430,"wal-append.run_cpu_time_us":33182,"wal-append.run_wall_time_us":2018153}]]}
I20260202 15:08:36.411708 14689 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.377607 14158 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:36.333333 14535 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.363960 14556 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.368104 14563 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.362807 14638 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:36.415722 14708 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:36.424716 12799 rpcz_store.cc:269] 0202 15:08:33.929875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.242820 (+2312945us) service_pool.cc:225] Handling call
0202 15:08:36.362439 (+119619us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.376436 14591 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:36.430500 12646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.051s sys 0.073s
W20260202 15:08:36.500250 12646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s user 0.052s sys 0.073s
I20260202 15:08:36.223688 14673 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:36.231323 13357 rpcz_store.cc:269] 0202 15:08:34.335413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.572183 (+236770us) service_pool.cc:225] Handling call
0202 15:08:36.154749 (+1582566us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.403154 14602 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.436179 13361 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:36.420961 13358 raft_consensus.cc:2410] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:36.408751 14341 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:36.412770 14316 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:36.237783 14706 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.450532 12722 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:36.454522 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:36.402825 14569 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.475589 12798 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:36.404394 14701 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.434629 14549 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.431970 14666 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.458812 12794 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
W20260202 15:08:36.431108 12791 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 258) took 3131 ms (client timeout 2108 ms). Trace:
I20260202 15:08:36.366969 14601 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.408872 14695 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.481585 14595 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:36.594043 12791 rpcz_store.cc:269] 0202 15:08:33.290452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.317908 (+ 27456us) service_pool.cc:225] Handling call
0202 15:08:36.407996 (+3090088us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b540036db54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:36.418849 (+ 10853us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":685106048}
I20260202 15:08:36.453684 14648 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:36.605062 14701 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:36.555693 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.476107 13354 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:36.455107 14536 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:36.500315 12796 raft_consensus.cc:1240] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:36.501724 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 264) took 2580 ms. Trace:
I20260202 15:08:36.639094 12800 rpcz_store.cc:276] 0202 15:08:33.921460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.476669 (+2555209us) service_pool.cc:225] Handling call
0202 15:08:36.501112 (+ 24443us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.501338 (+ 226us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:36.501376 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:36.501431 (+ 55us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:36.501473 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.501543 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:36.501660 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.507261 12721 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:36.458606 14503 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:36.451169 12795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:36.356282 14657 consensus_queue.cc:237] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.475162 14548 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.601523 14673 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:36.496658 12642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.011s sys 0.052s
W20260202 15:08:36.595508 14709 log.cc:927] Time spent T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20260202 15:08:36.635219 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1411) took 2675 ms. Trace:
I20260202 15:08:36.546625 12797 raft_consensus.cc:2393] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:36.496018 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:36.613153 14354 consensus_meta.cc:287] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.848s user 0.007s sys 0.004s
I20260202 15:08:36.509696 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1433) took 2536 ms. Trace:
I20260202 15:08:36.678752 12646 rpcz_store.cc:276] 0202 15:08:33.972925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:35.764317 (+1791392us) service_pool.cc:225] Handling call
0202 15:08:35.764636 (+ 319us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:35.764846 (+ 210us) raft_consensus.cc:1534] Early marking committed up to index 2
0202 15:08:35.836825 (+ 71979us) log.cc:844] Serialized 14 byte log entry
0202 15:08:36.509395 (+672570us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:36.509445 (+ 50us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:36.509481 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.509534 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:36.509638 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6881,"prepare.queue_time_us":833465,"prepare.run_wall_time_us":36,"raft.queue_time_us":134940,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":7291,"thread_start_us":686347,"threads_started":2}
I20260202 15:08:36.680630 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:36.681725 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:36.647603 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1402) took 2704 ms. Trace:
W20260202 15:08:36.648603 12642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s user 0.011s sys 0.053s
I20260202 15:08:36.645423 12795 raft_consensus.cc:606] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:36.683051 12796 rpcz_store.cc:276] 0202 15:08:33.943615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.476547 (+2532932us) service_pool.cc:225] Handling call
0202 15:08:36.500109 (+ 23562us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.647316 (+147207us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.647539 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.650108 12794 rpcz_store.cc:276] 0202 15:08:33.959856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.595896 (+2636040us) service_pool.cc:225] Handling call
0202 15:08:36.634591 (+ 38695us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.634808 (+ 217us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:36.634844 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:36.634924 (+ 80us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:36.634963 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.635024 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:36.635137 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.598536 14666 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:36.579880 14556 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:36.647011 14698 log.cc:927] Time spent T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.291s user 0.001s sys 0.000s
I20260202 15:08:36.507988 14638 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.684531 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:36.486691 14158 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.687204 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1737) took 2534 ms. Trace:
I20260202 15:08:36.707561 12800 rpcz_store.cc:276] 0202 15:08:34.152685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.640068 (+2487383us) service_pool.cc:225] Handling call
0202 15:08:36.686624 (+ 46556us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.686841 (+ 217us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:36.686874 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:36.686924 (+ 50us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:36.686962 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.687024 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:36.687147 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.593348 14705 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:36.688067 12726 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:36.433362 14596 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:36.720064 12724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 256) took 3768 ms (client timeout 1543 ms). Trace:
I20260202 15:08:36.720228 12718 raft_consensus.cc:1240] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
W20260202 15:08:36.720495 12724 rpcz_store.cc:269] 0202 15:08:32.952040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.952230 (+ 190us) service_pool.cc:225] Handling call
0202 15:08:36.719877 (+3767647us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400250dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:36.719992 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":346616064}
I20260202 15:08:36.524403 14711 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:36.510043 14591 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:36.690627 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1430) took 2718 ms. Trace:
I20260202 15:08:36.706240 12794 raft_consensus.cc:1275] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:36.692354 12649 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:36.707199 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:36.656165 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:36.710650 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35714 (request call id 13) took 2409 ms. Trace:
I20260202 15:08:36.685401 12791 raft_consensus.cc:1240] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:36.695693 14673 raft_consensus.cc:2749] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:36.769726 12795 rpcz_store.cc:276] 0202 15:08:34.296041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.451061 (+2155020us) service_pool.cc:225] Handling call
0202 15:08:36.705515 (+254454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.695276 14701 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:36.718650 12691 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.025s sys 0.016s
I20260202 15:08:36.712867 14158 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:36.774884 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1741) took 2582 ms. Trace:
I20260202 15:08:36.770677 12642 rpcz_store.cc:276] 0202 15:08:33.972313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:35.757537 (+1785224us) service_pool.cc:225] Handling call
0202 15:08:35.757845 (+ 308us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:35.758050 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 4
0202 15:08:35.973964 (+215914us) log.cc:844] Serialized 14 byte log entry
0202 15:08:36.689607 (+715643us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:36.689649 (+ 42us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:36.689687 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.689745 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:36.690571 (+ 826us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6304,"prepare.queue_time_us":759323,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":9649,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":8065,"thread_start_us":910995,"threads_started":2}
W20260202 15:08:36.776891 12615 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.036s sys 0.055s
I20260202 15:08:36.694348 14708 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:36.832587 12615 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s user 0.049s sys 0.081s
I20260202 15:08:36.706575 14476 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.835587 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:36.833132 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 637159us
I20260202 15:08:36.832671 12794 rpcz_store.cc:276] 0202 15:08:34.192270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.684342 (+2492072us) service_pool.cc:225] Handling call
0202 15:08:36.705914 (+ 21572us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.774566 (+ 68652us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.774803 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:36.814131 12691 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.726s user 0.025s sys 0.021s
I20260202 15:08:36.710644 13355 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:36.717588 14618 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:36.713732 14656 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.837999 14536 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:36.783802 12546 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.019s sys 0.025s
I20260202 15:08:36.694806 14666 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.716228 14459 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.748641 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:36.879297 12646 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:36.776793 13357 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 3
I20260202 15:08:36.774026 14695 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:36.724608 14514 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.776166 14563 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:36.859747 12546 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.688s user 0.019s sys 0.029s
I20260202 15:08:36.848874 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1403) took 2905 ms. Trace:
W20260202 15:08:36.861750 12615 connection.cc:232] RPC call timeout handler was delayed by -1.3922675860001164s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
I20260202 15:08:36.724855 14233 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.867025 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 718) took 1042 ms. Trace:
I20260202 15:08:36.816459 14644 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:36.816052 13356 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 709) took 2924 ms (client timeout 2661 ms). Trace:
W20260202 15:08:36.874876 12613 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.007s sys 0.009s
I20260202 15:08:36.815382 13358 raft_consensus.cc:2468] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:36.839375 14268 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.843057 14673 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:36.838923 14607 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:36.845391 14711 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:36.887451 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.888048 12642 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:36.778057 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:36.845978 14617 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:36.878394 12792 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.960s user 0.014s sys 0.020s
W20260202 15:08:36.899048 12615 connection.cc:232] RPC call timeout handler was delayed by -1.26626758600014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
I20260202 15:08:36.773636 14634 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.832760 14705 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.837682 14701 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:36.907231 12613 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.655s user 0.008s sys 0.010s
I20260202 15:08:36.851217 14689 consensus_queue.cc:1048] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260202 15:08:36.882860 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1738) took 2723 ms. Trace:
W20260202 15:08:36.913247 12690 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.055s sys 0.029s
W20260202 15:08:36.940821 12690 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s user 0.055s sys 0.029s
W20260202 15:08:36.941186 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 722986us
I20260202 15:08:36.890460 14569 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:36.915256 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1439) took 2925 ms. Trace:
I20260202 15:08:36.915376 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:36.915940 12763 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.016s sys 0.029s
I20260202 15:08:36.899248 14652 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:36.877398 14536 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.876125 14618 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.943287 12690 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:36.906081 14693 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:36.907086 13356 rpcz_store.cc:269] 0202 15:08:33.852603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.855858 (+ 3255us) service_pool.cc:225] Handling call
0202 15:08:36.724743 (+2868885us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b540048f654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:36.777387 (+ 52644us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67955968}
W20260202 15:08:36.921583 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 542072us
I20260202 15:08:36.853207 14476 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:36.918109 12688 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.703s user 0.006s sys 0.034s
I20260202 15:08:36.928845 12764 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:36.915427 14673 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.918164 14706 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:36.904552 13355 rpcz_store.cc:276] 0202 15:08:35.817816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.448282 (+630466us) service_pool.cc:225] Handling call
0202 15:08:36.860066 (+411784us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:36.965910 12763 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.760s user 0.017s sys 0.031s
I20260202 15:08:36.923599 14548 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:36.923498 14601 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:36.932646 14293 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:36.898968 12791 rpcz_store.cc:276] 0202 15:08:33.943929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.595318 (+2651389us) service_pool.cc:225] Handling call
0202 15:08:36.636523 (+ 41205us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.776636 (+140113us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.845787 (+ 69151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.964767 12642 rpcz_store.cc:276] 0202 15:08:33.990355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.887031 (+2896676us) service_pool.cc:225] Handling call
0202 15:08:36.887903 (+ 872us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.915052 (+ 27149us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.915212 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:36.935386 13354 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:36.970510 14715 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:36.943511 14704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.000s sys 0.024s
W20260202 15:08:36.972610 12688 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.778s user 0.007s sys 0.034s
W20260202 15:08:36.972934 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 44181.4us
W20260202 15:08:36.934053 12792 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.069s user 0.016s sys 0.021s
W20260202 15:08:36.941828 14645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.000s sys 0.028s
I20260202 15:08:36.973883 12690 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:36.931877 14711 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:36.982496 12614 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.013s sys 0.023s
W20260202 15:08:36.988754 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 782531us
I20260202 15:08:36.995947 12642 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1c6861781ee34db9b79870494af1cced"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:36.985776 14638 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:36.969048 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.982326 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:36.945150 14644 consensus_queue.cc:1048] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260202 15:08:36.987426 14693 raft_consensus.cc:2804] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:36.987248 14476 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.992707 14725 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:37.083911 12614 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.721s user 0.020s sys 0.026s
I20260202 15:08:36.940557 12793 rpcz_store.cc:276] 0202 15:08:34.159873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.653245 (+2493372us) service_pool.cc:225] Handling call
0202 15:08:36.724232 (+ 70987us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.836332 (+112100us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:36.839952 (+ 3620us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:36.843278 (+ 3326us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:36.849725 (+ 6447us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:36.859743 (+ 10018us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:36.881357 (+ 21614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.084640 14601 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:37.082665 12642 raft_consensus.cc:606] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
W20260202 15:08:37.085178 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 722081us
I20260202 15:08:37.088261 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:37.071250 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 1.23287e+06us
W20260202 15:08:37.090128 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 116360us
I20260202 15:08:37.094245 12690 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:37.095470 12614 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.084064 14725 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:36.993784 14591 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.111745 12763 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:37.119489 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:37.100688 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:37.085287 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:37.174230 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 78833.7us
I20260202 15:08:37.131194 12794 raft_consensus.cc:2393] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:08:37.005522 14704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s user 0.000s sys 0.025s
I20260202 15:08:37.173440 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.178318 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:36.999202 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.186472 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1438) took 3209 ms. Trace:
I20260202 15:08:37.188195 12642 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
W20260202 15:08:37.060465 14641 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.015s sys 0.023s
W20260202 15:08:37.083375 14645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.775s user 0.001s sys 0.034s
I20260202 15:08:37.189361 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.075538 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:37.187033 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 248745us
I20260202 15:08:37.188320 12646 rpcz_store.cc:276] 0202 15:08:33.977012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.886038 (+2909026us) service_pool.cc:225] Handling call
0202 15:08:36.886433 (+ 395us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.886627 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:08:36.904975 (+ 18348us) log.cc:844] Serialized 14 byte log entry
0202 15:08:37.186064 (+281089us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:37.186115 (+ 51us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:37.186169 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.186256 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.186411 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55966,"prepare.queue_time_us":14672,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":2358,"raft.queue_time_us":56816,"thread_start_us":224737,"threads_started":1}
I20260202 15:08:37.261207 12794 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
W20260202 15:08:37.260447 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 68800us
I20260202 15:08:37.095156 14596 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.269924 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:37.111181 14715 raft_consensus.cc:2749] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:37.007714 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.270053 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.269169 12642 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:37.006938 14666 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.082532 14693 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:37.301968 12688 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.302114 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.131596 14617 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260202 15:08:37.304966 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1419) took 4537 ms (client timeout 2600 ms). Trace:
I20260202 15:08:37.000409 14293 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.131223 14268 consensus_queue.cc:1059] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.146s
W20260202 15:08:37.305390 12644 rpcz_store.cc:269] 0202 15:08:32.765365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.766654 (+ 1289us) service_pool.cc:225] Handling call
0202 15:08:37.299943 (+4533289us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001decd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.300036 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2329859840}
I20260202 15:08:37.173066 14494 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:37.125684 13358 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
W20260202 15:08:37.260949 14641 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.854s user 0.019s sys 0.035s
W20260202 15:08:37.270742 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 201542us
W20260202 15:08:37.270829 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 92975.6us
I20260202 15:08:37.299458 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.189636 14354 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:37.292176 13326 leader_election.cc:336] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Timed out: RequestConsensusVote RPC to 127.11.46.193:40057 timed out after 1.609s (SENT)
I20260202 15:08:37.303689 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.359061 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:37.274080 12799 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.044s sys 0.054s
W20260202 15:08:37.359760 12799 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s user 0.053s sys 0.066s
I20260202 15:08:37.260800 14708 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:37.307825 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.085181 14601 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:37.176016 14656 consensus_queue.cc:1059] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.125s
I20260202 15:08:37.317833 14725 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:37.318886 14621 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.320036 14618 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.320322 14645 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:37.320932 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 531) took 3533 ms. Trace:
I20260202 15:08:37.276721 13355 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:37.322338 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.322943 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1716) took 5609 ms. Trace:
I20260202 15:08:37.316730 14715 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:37.189430 14724 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
W20260202 15:08:37.323609 12647 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1429) took 3563 ms (client timeout 3287 ms). Trace:
I20260202 15:08:37.270367 12793 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:37.329970 14705 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.333338 14476 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.344252 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 499) took 1529 ms. Trace:
I20260202 15:08:37.344405 14268 raft_consensus.cc:3055] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:37.344504 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1651) took 2977 ms. Trace:
I20260202 15:08:37.309391 14617 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:37.309989 14591 consensus_queue.cc:1059] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
W20260202 15:08:37.349436 12726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1345) took 4267 ms (client timeout 3374 ms). Trace:
I20260202 15:08:37.349881 14649 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.350242 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:37.350682 12649 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1447) took 1570 ms (client timeout 1885 ms). Trace:
I20260202 15:08:37.267015 13356 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:37.273402 14619 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:37.351632 13361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 500) took 1536 ms (client timeout 1772 ms). Trace:
W20260202 15:08:37.356205 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1437) took 3470 ms (client timeout 2661 ms). Trace:
I20260202 15:08:37.352205 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1430) took 3592 ms. Trace:
I20260202 15:08:37.352743 13328 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:37.360558 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1735) took 3264 ms. Trace:
W20260202 15:08:37.354631 14176 consensus_meta.cc:287] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.175s user 0.010s sys 0.000s
I20260202 15:08:37.299945 14548 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:37.306267 14693 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:37.515491 13363 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:37.370515 13326 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:37.515733 12647 rpcz_store.cc:269] 0202 15:08:33.759825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.308225 (+548400us) service_pool.cc:225] Handling call
0202 15:08:37.323462 (+3015237us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b540007a3d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.323551 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1246767872}
I20260202 15:08:37.314963 14494 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:37.516251 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 163556us
I20260202 15:08:37.366496 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.386283 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.408113 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.424809 12689 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:37.515100 12645 rpcz_store.cc:269] 0202 15:08:33.886249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.191400 (+305151us) service_pool.cc:225] Handling call
0202 15:08:37.341264 (+3149864us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b540029a254. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.356112 (+ 14848us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1306033024}
W20260202 15:08:37.517625 12726 rpcz_store.cc:269] 0202 15:08:33.081848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.259731 (+177883us) service_pool.cc:225] Handling call
0202 15:08:37.337390 (+4077659us) spinlock_profiling.cc:243] Waited 3.39 s on lock 0x7b5400199e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.349374 (+ 11984us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7787430656}
W20260202 15:08:37.517283 13361 rpcz_store.cc:269] 0202 15:08:35.815491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.215531 (+400040us) service_pool.cc:225] Handling call
0202 15:08:37.344260 (+1128729us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54005eddd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.351565 (+ 7305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":441607424}
I20260202 15:08:37.515254 13360 rpcz_store.cc:276] 0202 15:08:33.787312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.787449 (+ 137us) service_pool.cc:225] Handling call
0202 15:08:33.787690 (+ 241us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:33.836522 (+ 48832us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:33.836578 (+ 56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:33.836627 (+ 49us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:33.836680 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.320745 (+3484065us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b5400264b54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:37.320784 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.320886 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86824832}
I20260202 15:08:37.516958 12641 rpcz_store.cc:276] 0202 15:08:33.760132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.375606 (+615474us) service_pool.cc:225] Handling call
0202 15:08:34.445329 (+ 69723us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.222391 (+1777062us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.343198 (+1120807us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b5400280854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:37.352139 (+ 8941us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3997881088}
I20260202 15:08:37.531620 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:37.525446 13363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 535) took 3209 ms (client timeout 1653 ms). Trace:
W20260202 15:08:37.536916 13363 rpcz_store.cc:269] 0202 15:08:34.316301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.316403 (+ 102us) service_pool.cc:225] Handling call
0202 15:08:37.336986 (+3020583us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:37.525377 (+188391us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3328256}
I20260202 15:08:37.537884 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:37.537014 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 244864us
I20260202 15:08:37.527413 12646 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 1.
I20260202 15:08:37.527113 12794 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:37.516935 12724 rpcz_store.cc:276] 0202 15:08:34.366953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.769320 (+2402367us) service_pool.cc:225] Handling call
0202 15:08:36.769626 (+ 306us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:36.776186 (+ 6560us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:36.776214 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:36.776899 (+ 685us) log.cc:844] Serialized 26 byte log entry
0202 15:08:37.047371 (+270472us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:37.047419 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.047501 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:37.344326 (+296825us) raft_consensus.cc:1685] finished
0202 15:08:37.344344 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.344454 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2404,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":58338,"thread_start_us":188192,"threads_started":1,"wal-append.queue_time_us":468000}
I20260202 15:08:37.350888 14701 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.355787 14616 raft_consensus.cc:3055] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:37.555645 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:37.471371 14673 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.447407 14481 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:37.557757 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.576659 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:37.577116 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.575790 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.577467 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:37.555408 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 253498us
W20260202 15:08:37.572240 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 147489us
I20260202 15:08:37.557533 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:37.573300 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1650) took 3229 ms (client timeout 1526 ms). Trace:
I20260202 15:08:37.578074 13363 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
W20260202 15:08:37.573292 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:37.578819 12641 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:37.578683 12689 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2
W20260202 15:08:37.578622 12721 rpcz_store.cc:269] 0202 15:08:34.344024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.366182 (+ 22158us) service_pool.cc:225] Handling call
0202 15:08:37.573148 (+3206966us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b5400157f54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.573244 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4590878976}
I20260202 15:08:37.579746 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:37.353648 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1358) took 3178 ms (client timeout 1932 ms). Trace:
I20260202 15:08:37.311635 14666 consensus_queue.cc:1048] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.224s
I20260202 15:08:37.425084 14536 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.515460 14268 raft_consensus.cc:740] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:37.575101 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.575397 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.584256 12644 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:37.367497 14535 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:37.584631 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 512) took 1341 ms. Trace:
I20260202 15:08:37.584784 12689 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.584256 12688 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.525662 14617 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:37.584975 13361 rpcz_store.cc:276] 0202 15:08:36.242934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.529540 (+1286606us) service_pool.cc:225] Handling call
0202 15:08:37.584583 (+ 55043us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.533803 14494 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:37.529409 13356 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 498) took 1714 ms (client timeout 2249 ms). Trace:
I20260202 15:08:37.536728 14725 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:37.516997 12799 rpcz_store.cc:276] 0202 15:08:34.096125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.595991 (+2499866us) service_pool.cc:225] Handling call
0202 15:08:36.638214 (+ 42223us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:36.648648 (+ 10434us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:36.648685 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:36.678411 (+ 29726us) log.cc:844] Serialized 26 byte log entry
0202 15:08:37.360172 (+681761us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:37.360219 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.360312 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:37.360364 (+ 52us) raft_consensus.cc:1685] finished
0202 15:08:37.360380 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.360500 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6372,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":10927,"spinlock_wait_cycles":767104,"thread_start_us":589812,"threads_started":1,"wal-append.queue_time_us":635987}
W20260202 15:08:37.517645 12649 rpcz_store.cc:269] 0202 15:08:35.780111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.434171 (+654060us) service_pool.cc:225] Handling call
0202 15:08:37.339774 (+905603us) spinlock_profiling.cc:243] Waited 95.7 ms on lock 0x7b540008acd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.350628 (+ 10854us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":220084480}
I20260202 15:08:37.541611 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.556551 13360 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:37.593401 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.593824 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 511) took 1351 ms. Trace:
I20260202 15:08:37.580873 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 514) took 1272 ms. Trace:
I20260202 15:08:37.594324 13360 rpcz_store.cc:276] 0202 15:08:36.242642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.530222 (+1287580us) service_pool.cc:225] Handling call
0202 15:08:37.593729 (+ 63507us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.594843 13363 rpcz_store.cc:276] 0202 15:08:36.307993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.537830 (+1229837us) service_pool.cc:225] Handling call
0202 15:08:37.580818 (+ 42988us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.524818 14727 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:37.581765 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 53677.8us
I20260202 15:08:37.583093 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.538810 14704 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:37.537791 14476 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:37.524613 14715 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:37.578166 12794 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 1.
I20260202 15:08:37.591197 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:37.591598 12690 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:37.602151 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.515208 13358 rpcz_store.cc:276] 0202 15:08:35.815189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.923388 (+1108199us) service_pool.cc:225] Handling call
0202 15:08:37.344178 (+420790us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.603336 13361 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:37.603865 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 546) took 1229 ms. Trace:
I20260202 15:08:37.604238 13361 rpcz_store.cc:276] 0202 15:08:36.375627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.587000 (+1211373us) service_pool.cc:225] Handling call
0202 15:08:37.603809 (+ 16809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.385082 14708 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.382668 14354 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.573451 14616 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:37.596014 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.612286 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:37.557921 14724 raft_consensus.cc:697] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:37.597041 12764 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.597616 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.596575 12724 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:37.613230 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.598836 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.528786 14548 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:37.586570 14676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.171s user 0.002s sys 0.021s
I20260202 15:08:37.600335 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.631614 12690 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.595392 12615 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.605566 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.533470 14176 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.583750 14618 raft_consensus.cc:2749] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:37.586410 14482 raft_consensus.cc:2764] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260202 15:08:37.529495 14619 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:37.631098 14676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.225s user 0.002s sys 0.021s
I20260202 15:08:37.633931 14636 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:37.635522 14595 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:37.652558 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1424) took 3888 ms. Trace:
W20260202 15:08:37.653405 14703 log.cc:927] Time spent T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 1.746s user 0.003s sys 0.000s
I20260202 15:08:37.654981 12614 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.655133 14548 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.655467 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:37.656175 12648 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:37.656838 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.657239 14708 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.657683 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.611285 12763 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.613801 13360 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:37.583241 14481 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.516521 12792 rpcz_store.cc:276] 0202 15:08:31.700769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:31.703224 (+ 2455us) service_pool.cc:225] Handling call
0202 15:08:31.719340 (+ 16116us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:31.719555 (+ 215us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:31.719581 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:31.719983 (+ 402us) log.cc:844] Serialized 26 byte log entry
0202 15:08:32.072594 (+352611us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:32.664603 (+592009us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b18004894c0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea4b12d3 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad 0000000000379254 0000000000379099 00007f24ea49469a
0202 15:08:35.818291 (+3153688us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b18004894c0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ed10685c 00007f24ea4942eb 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4ae335 00007f24f0f59317 00007f24ed569627 00007f24ed578d3f
0202 15:08:35.829326 (+ 11035us) log.cc:844] Serialized 14 byte log entry
0202 15:08:37.265062 (+1435736us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.273431 (+ 8369us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:37.294347 (+ 20916us) raft_consensus.cc:1685] finished
0202 15:08:37.294371 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.309651 (+ 15280us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73160,"prepare.queue_time_us":225387,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":85344,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":12003,"spinlock_wait_cycles":39240192,"thread_start_us":1737097,"threads_started":3,"wal-append.queue_time_us":1749758,"wal-append.run_cpu_time_us":1630,"wal-append.run_wall_time_us":1006677}
I20260202 15:08:37.614961 14724 consensus_queue.cc:237] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.615962 14616 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.616776 14150 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:37.617359 14601 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:37.580746 12717 rpcz_store.cc:269] 0202 15:08:34.175643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.182111 (+ 6468us) service_pool.cc:225] Handling call
0202 15:08:37.353472 (+3171361us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54002ffb54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.353582 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3336444288}
I20260202 15:08:37.596750 14657 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:37.618371 14737 raft_consensus.cc:2749] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260202 15:08:37.618774 14663 log.cc:927] Time spent T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 3.045s user 0.000s sys 0.004s
I20260202 15:08:37.618973 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.619072 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.620468 14354 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.620923 12721 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:37.621417 13357 consensus_meta.cc:287] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.718s user 0.002s sys 0.005s
I20260202 15:08:37.590961 14293 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.536378 14645 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.597839 14591 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:37.631625 12644 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
W20260202 15:08:37.668318 14700 log.cc:927] Time spent T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 1.055s user 0.002s sys 0.000s
I20260202 15:08:37.661739 14486 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:37.636300 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.637006 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.600605 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.638002 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:37.591601 13356 rpcz_store.cc:269] 0202 15:08:35.814877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.985265 (+1170388us) service_pool.cc:225] Handling call
0202 15:08:37.524767 (+539502us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.635303 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1348) took 4511 ms. Trace:
I20260202 15:08:37.711131 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.606804 14715 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:37.639153 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:37.639401 14735 log.cc:927] Time spent T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260202 15:08:37.639338 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.608953 14476 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.641144 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:37.644827 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 49517.5us
I20260202 15:08:37.719247 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.720041 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:37.646675 14711 consensus_meta.cc:287] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.522s user 0.000s sys 0.007s
I20260202 15:08:37.720568 12647 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
W20260202 15:08:37.648531 14678 consensus_meta.cc:287] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.273s user 0.001s sys 0.009s
I20260202 15:08:37.608853 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:37.652962 12643 rpcz_store.cc:276] 0202 15:08:33.763650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.675458 (+911808us) service_pool.cc:225] Handling call
0202 15:08:34.854203 (+178745us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.328878 (+1474675us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.652296 (+1323418us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b54000127d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:37.652486 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2238024960}
W20260202 15:08:37.678898 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 83445.9us
W20260202 15:08:37.667650 12725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1364) took 2815 ms (client timeout 3517 ms). Trace:
I20260202 15:08:37.658213 12648 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
W20260202 15:08:37.739651 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:37.700981 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 506) took 1841 ms. Trace:
W20260202 15:08:37.701730 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 90522.3us
I20260202 15:08:37.740142 13360 rpcz_store.cc:276] 0202 15:08:35.859381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.598627 (+1739246us) service_pool.cc:225] Handling call
0202 15:08:37.700922 (+102295us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.741299 12763 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.741209 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.741873 13360 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:37.702171 13363 raft_consensus.cc:2410] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:37.670316 12795 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:37.743304 12763 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.702956 14617 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:37.743721 14678 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:37.745590 14619 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:37.746574 14617 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.597764 13328 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:37.747648 14678 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.606191 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.748445 12726 raft_consensus.cc:2393] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:37.749111 12724 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:37.704824 12764 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:37.658156 14176 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:37.633172 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.705324 14683 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:37.754022 14683 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:37.754427 14683 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:37.756819 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1740) took 3569 ms. Trace:
I20260202 15:08:37.701311 14657 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:37.659016 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1644) took 4525 ms (client timeout 4806 ms). Trace:
I20260202 15:08:37.714280 12721 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:37.714651 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.758994 14150 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:37.759646 12646 raft_consensus.cc:2393] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:37.751796 14673 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:37.760232 14673 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:37.760568 14673 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:37.715907 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.717433 12642 raft_consensus.cc:2410] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260202 15:08:37.717485 12649 raft_consensus.cc:2393] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:37.761488 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.761708 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.761967 12644 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:37.762427 12649 raft_consensus.cc:2393] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:37.708217 14616 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:37.711014 12720 rpcz_store.cc:276] 0202 15:08:33.122054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.707295 (+585241us) service_pool.cc:225] Handling call
0202 15:08:33.708656 (+ 1361us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:33.731720 (+ 23064us) raft_consensus.cc:1534] Early marking committed up to index 2
0202 15:08:33.776677 (+ 44957us) log.cc:844] Serialized 14 byte log entry
0202 15:08:37.346358 (+3569681us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b1800385240. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe5da 00007f24ed5696aa
0202 15:08:37.600790 (+254432us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:37.608500 (+ 7710us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:37.613780 (+ 5280us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.631199 (+ 17419us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.633774 (+ 2575us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33744,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":1641,"raft.queue_time_us":2831,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":6777,"spinlock_wait_cycles":27084672,"thread_start_us":242245,"threads_started":1,"wal-append.queue_time_us":3866507}
I20260202 15:08:37.762778 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.704018 13357 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:37.738332 14711 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:37.763619 13357 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 503) took 1934 ms (client timeout 2032 ms). Trace:
W20260202 15:08:37.763975 13357 rpcz_store.cc:269] 0202 15:08:35.828924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.631477 (+802553us) service_pool.cc:225] Handling call
0202 15:08:37.763555 (+1132078us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.759049 14657 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:37.723258 12798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1407) took 3774 ms (client timeout 1880 ms). Trace:
I20260202 15:08:37.723522 14681 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.725693 14494 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:37.766891 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.768266 13357 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:37.769146 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.769731 13357 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 1.
I20260202 15:08:37.715528 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.718838 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:37.774305 14742 log.cc:927] Time spent T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260202 15:08:37.728802 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1433) took 3965 ms. Trace:
I20260202 15:08:37.774654 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:37.776111 14645 leader_election.cc:290] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.774667 12650 rpcz_store.cc:276] 0202 15:08:33.763376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.383403 (+620027us) service_pool.cc:225] Handling call
0202 15:08:34.444469 (+ 61066us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.421087 (+1976618us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.728619 (+1307532us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b540020d354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:37.728757 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4393815936}
I20260202 15:08:37.728606 12723 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:37.722651 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.665879 14693 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.735495 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.777995 14604 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:37.779119 12799 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:37.779166 14604 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.779973 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.780121 14693 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.766196 13355 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:37.720000 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.781515 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.782091 12717 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:37.782881 13356 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:37.783370 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:37.783743 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.784188 14570 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.784408 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.785406 14468 consensus_queue.cc:1048] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:08:37.789866 14559 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.725795 14476 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.737339 14688 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.791801 14656 raft_consensus.cc:3055] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:37.792258 14656 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:37.792774 13354 consensus_meta.cc:287] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.722s user 0.006s sys 0.003s
I20260202 15:08:37.793102 13354 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
W20260202 15:08:37.793627 13354 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 545) took 1421 ms (client timeout 1634 ms). Trace:
I20260202 15:08:37.793139 14656 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:37.793954 13354 rpcz_store.cc:269] 0202 15:08:36.371848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.687122 (+315274us) service_pool.cc:225] Handling call
0202 15:08:37.793576 (+1106454us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:37.794888 13355 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 542) took 1436 ms (client timeout 1718 ms). Trace:
W20260202 15:08:37.795226 13355 rpcz_store.cc:269] 0202 15:08:36.358750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.995806 (+637056us) service_pool.cc:225] Handling call
0202 15:08:37.794824 (+799018us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.796175 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.797370 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:37.708828 14591 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:37.739760 12725 rpcz_store.cc:269] 0202 15:08:34.852404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.855505 (+ 3101us) service_pool.cc:225] Handling call
0202 15:08:37.667454 (+2811949us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54003007d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.667589 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":554035072}
I20260202 15:08:37.801862 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.763358 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.802035 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.746622 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:37.747570 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 149861us
I20260202 15:08:37.802552 12725 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:37.803232 13328 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:37.596397 14725 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.750419 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.803984 13328 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.744441 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:37.673956 14722 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260202 15:08:37.751334 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:37.757045 12689 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37626 dropped due to backpressure: service queue is full with 50 items
I20260202 15:08:37.709641 14564 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:37.758229 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:37.757305 12796 rpcz_store.cc:276] 0202 15:08:34.187766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.683929 (+2496163us) service_pool.cc:225] Handling call
0202 15:08:37.335957 (+652028us) spinlock_profiling.cc:243] Waited 6.93 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:37.350020 (+ 14063us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:37.556197 (+206177us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:37.556242 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:37.573664 (+ 17422us) log.cc:844] Serialized 26 byte log entry
0202 15:08:37.624677 (+ 51013us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:37.624715 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.756420 (+131705us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540004c754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:37.756496 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:37.756569 (+ 73us) raft_consensus.cc:1685] finished
0202 15:08:37.756589 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.756750 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":85,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":18566,"spinlock_wait_cycles":489279616,"thread_start_us":28561,"threads_started":1,"wal-append.queue_time_us":68983}
W20260202 15:08:37.752867 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 102028us
W20260202 15:08:37.759109 12719 rpcz_store.cc:269] 0202 15:08:33.133401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.688467 (+555066us) service_pool.cc:225] Handling call
0202 15:08:37.658787 (+3970320us) spinlock_profiling.cc:243] Waited 2.53 s on lock 0x7b5400144954. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.658957 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5811392768}
I20260202 15:08:37.759537 14150 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:37.811455 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.811697 14150 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:37.812451 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.812953 12796 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:37.813098 14725 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:37.806762 14722 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:37.813686 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.715716 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.807834 14476 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:37.814266 14476 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:37.814586 14476 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:37.814800 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:37.815363 12641 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:37.815436 14481 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:37.669656 12797 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 263) took 3772 ms (client timeout 1973 ms). Trace:
W20260202 15:08:37.818851 12797 rpcz_store.cc:269] 0202 15:08:33.897256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:35.764436 (+1867180us) service_pool.cc:225] Handling call
0202 15:08:37.669418 (+1904982us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54002eb654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.669579 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":685250304}
I20260202 15:08:37.819943 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.819970 14150 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.821051 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.821849 14150 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.809718 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.760463 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.824079 12646 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:37.803613 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.722985 13362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 536) took 3394 ms. Trace:
I20260202 15:08:37.823663 14564 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:37.826507 13359 raft_consensus.cc:2410] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260202 15:08:37.726694 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1646) took 3405 ms. Trace:
I20260202 15:08:37.827380 12718 rpcz_store.cc:276] 0202 15:08:34.320794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.321518 (+ 724us) service_pool.cc:225] Handling call
0202 15:08:36.720032 (+2398514us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:36.771392 (+ 51360us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.726510 (+955118us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b5400250dd0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.726640 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5483276032}
I20260202 15:08:37.770486 12794 raft_consensus.cc:2368] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3.
I20260202 15:08:37.771939 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.769227 14494 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:37.774189 12793 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:37.758772 14619 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:37.827788 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.778107 12647 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:37.829702 12726 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:37.830426 12647 raft_consensus.cc:1275] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:37.830502 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.831070 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1436) took 3026 ms. Trace:
I20260202 15:08:37.831115 12726 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:37.831444 12647 rpcz_store.cc:276] 0202 15:08:34.804860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.829870 (+3025010us) service_pool.cc:225] Handling call
0202 15:08:37.830149 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:37.830874 (+ 725us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.831028 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.831943 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.832515 12726 raft_consensus.cc:2393] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:37.832767 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1437) took 3027 ms. Trace:
I20260202 15:08:37.833180 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.778128 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 291) took 3392 ms. Trace:
I20260202 15:08:37.833990 12726 raft_consensus.cc:2393] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:37.833874 12650 rpcz_store.cc:276] 0202 15:08:34.385708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.777462 (+3391754us) service_pool.cc:225] Handling call
0202 15:08:37.777716 (+ 254us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:37.777864 (+ 148us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:37.777886 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:37.777924 (+ 38us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:37.777950 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.777994 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.778074 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.780475 12799 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:37.831557 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.835171 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.835740 12793 raft_consensus.cc:2393] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:37.835803 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.826058 14564 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:37.836642 12799 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:37.836596 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.837517 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.838032 12799 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:37.785626 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.636794 14268 consensus_queue.cc:260] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:37.827214 13359 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 502) took 1999 ms (client timeout 1809 ms). Trace:
W20260202 15:08:37.839130 13359 rpcz_store.cc:269] 0202 15:08:35.827438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.357881 (+530443us) service_pool.cc:225] Handling call
0202 15:08:37.807573 (+1449692us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:37.827134 (+ 19561us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9847552}
I20260202 15:08:37.839985 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1414) took 1594 ms. Trace:
I20260202 15:08:37.840313 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.840286 12795 rpcz_store.cc:276] 0202 15:08:36.245954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.776568 (+530614us) service_pool.cc:225] Handling call
0202 15:08:37.670104 (+893536us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:37.743011 (+ 72907us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.839814 (+ 96803us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b54002eb650. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.839938 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2054343552}
I20260202 15:08:37.840905 13354 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:37.841310 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.841491 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:37.841678 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.841770 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.842276 12726 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:37.842686 12646 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:37.843082 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.843312 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1448) took 1996 ms. Trace:
I20260202 15:08:37.843672 12646 rpcz_store.cc:276] 0202 15:08:35.847391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.842153 (+1994762us) service_pool.cc:225] Handling call
0202 15:08:37.842420 (+ 267us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:37.843101 (+ 681us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.843268 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.736569 14636 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:37.770574 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.805891 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:37.835803 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.806517 12648 raft_consensus.cc:2410] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260202 15:08:37.808099 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:37.847754 12718 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
W20260202 15:08:37.808238 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 151970us
I20260202 15:08:37.808399 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:37.794589 14656 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:37.809835 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:37.810460 12764 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.784278 13356 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:37.809791 12720 raft_consensus.cc:2393] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:37.810617 12645 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:37.802899 14591 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:37.813813 14722 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:37.718824 13358 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:37.763350 12644 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:37.825076 13362 rpcz_store.cc:276] 0202 15:08:34.328863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:34.329005 (+ 142us) service_pool.cc:225] Handling call
0202 15:08:34.367683 (+ 38678us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:34.817576 (+449893us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:34.817606 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:34.817643 (+ 37us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:34.817670 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.722751 (+2905081us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b5400511654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:37.722818 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.722936 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1034224384}
W20260202 15:08:37.766640 12798 rpcz_store.cc:269] 0202 15:08:33.949348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:35.673207 (+1723859us) service_pool.cc:225] Handling call
0202 15:08:37.723100 (+2049893us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54001f7cd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.723195 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":739857664}
I20260202 15:08:37.826455 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1440) took 3819 ms. Trace:
I20260202 15:08:37.828974 13363 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:37.887490 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 497) took 2073 ms. Trace:
I20260202 15:08:37.887835 13363 rpcz_store.cc:276] 0202 15:08:35.814403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.752176 (+1937773us) service_pool.cc:225] Handling call
0202 15:08:37.887423 (+135247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.833081 12647 rpcz_store.cc:276] 0202 15:08:34.805047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.832158 (+3027111us) service_pool.cc:225] Handling call
0202 15:08:37.832417 (+ 259us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:37.832557 (+ 140us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:37.832581 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:37.832603 (+ 22us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:37.832635 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.832667 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.832730 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.837844 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.809376 12689 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37626 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1667)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 282)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1393)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1660)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 280)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 284)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 281)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 278)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1662)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 273)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1362)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1363)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1366)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1652)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1654)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1375)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1379)
I20260202 15:08:37.841820 12795 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:37.804675 13360 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 3
I20260202 15:08:37.732445 14715 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:37.770937 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:37.800000 12765 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.829207 12796 raft_consensus.cc:2393] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:37.809206 14559 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:37.849432 12688 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:37.849162 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.867524 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1653) took 1008 ms. Trace:
I20260202 15:08:37.851361 12614 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.899170 12726 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:37.869000 14711 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.842680 13354 raft_consensus.cc:2410] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260202 15:08:37.870986 13328 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:37.839032 14268 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:37.887166 12643 rpcz_store.cc:276] 0202 15:08:34.007045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.775583 (+3768538us) service_pool.cc:225] Handling call
0202 15:08:37.775906 (+ 323us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:37.822863 (+ 46957us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:37.822895 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:37.822941 (+ 46us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:37.822969 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.826157 (+ 3188us) spinlock_profiling.cc:243] Waited 34.7 ms on lock 0x7b54001df6d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:37.826226 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.826394 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":79799552}
I20260202 15:08:37.906147 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.838567 12717 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:37.871655 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:37.868571 12641 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:37.872329 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:37.911628 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.872709 12763 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.873718 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:37.841312 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.848830 14636 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:37.878418 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1439) took 1648 ms. Trace:
I20260202 15:08:37.913252 12646 rpcz_store.cc:276] 0202 15:08:36.229547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.864085 (+1634538us) service_pool.cc:225] Handling call
0202 15:08:37.877578 (+ 13493us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:37.877845 (+ 267us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:37.877877 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:37.877972 (+ 95us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:37.878031 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.878165 (+ 134us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.878317 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.890457 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.897001 12645 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:37.855456 13326 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:37.914530 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 65191.5us
I20260202 15:08:37.914783 12795 raft_consensus.cc:2393] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:08:37.915057 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 59719.8us
I20260202 15:08:37.892601 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1367) took 1044 ms. Trace:
I20260202 15:08:37.900878 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 516) took 1518 ms. Trace:
I20260202 15:08:37.931533 13357 rpcz_store.cc:276] 0202 15:08:36.379171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.770346 (+1391175us) service_pool.cc:225] Handling call
0202 15:08:37.897876 (+127530us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.900005 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.904891 13360 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:37.932415 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.932889 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:37.908015 13328 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:37.933552 13360 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:37.934216 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:37.935869 13328 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.908885 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.936910 12798 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:37.908054 12647 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:37.912142 13363 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:37.912568 12718 raft_consensus.cc:2393] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:37.843545 14673 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.915539 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.934279 12726 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:37.898312 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:37.916781 12688 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:37.899307 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 48040.8us
I20260202 15:08:37.916720 13326 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.943787 12614 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:37.943907 12688 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.912786 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.899083 12725 rpcz_store.cc:276] 0202 15:08:36.859182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.803150 (+943968us) service_pool.cc:225] Handling call
0202 15:08:37.867476 (+ 64326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.918802 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.946326 13326 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.946578 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.947360 12795 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:37.905700 13355 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:37.919385 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.919533 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.920264 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.920879 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.918792 13359 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:37.921568 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.915143 14725 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:37.922974 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.923307 14657 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:37.959820 13354 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:37.960711 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.961230 13354 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:37.961318 14657 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:37.904553 14660 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:37.961980 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.962028 14660 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:37.910897 14559 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:37.918771 14711 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:37.913147 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:37.963243 14711 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:37.963461 13356 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:37.963774 14711 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:37.927701 14715 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:37.929690 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34778 (request call id 1422) took 3073 ms. Trace:
I20260202 15:08:37.964818 12646 rpcz_store.cc:276] 0202 15:08:34.855903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.914108 (+3058205us) service_pool.cc:225] Handling call
0202 15:08:37.929638 (+ 15530us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.919080 14683 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:37.950836 14649 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:37.966955 14649 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:37.967307 14649 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:37.967665 14683 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:37.939357 12721 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:37.939965 13328 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.944876 12614 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:37.938629 13357 raft_consensus.cc:2376] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3. Re-sending same reply.
I20260202 15:08:37.951617 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.951000 14616 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:37.951956 14656 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:37.953086 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.954125 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.952711 14740 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:37.956816 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.957574 13361 raft_consensus.cc:2393] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:37.958220 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:37.958859 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.918574 12720 rpcz_store.cc:276] 0202 15:08:36.848216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.763127 (+914911us) service_pool.cc:225] Handling call
0202 15:08:37.887297 (+124170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:37.959228 12796 raft_consensus.cc:2410] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260202 15:08:37.945706 14701 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:37.948268 14176 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:37.975118 13354 raft_consensus.cc:2410] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:37.970458 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:38.031798 12717 raft_consensus.cc:2393] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:37.959237 12793 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:38.033167 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.969069 14725 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:37.951709 14613 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:37.971727 12688 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:38.038180 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 66512.8us
I20260202 15:08:37.951419 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.972323 13328 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:37.968220 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 509) took 2089 ms. Trace:
I20260202 15:08:37.953665 14676 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:37.952744 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.975019 14535 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:37.975975 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.954833 14481 consensus_queue.cc:1059] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260202 15:08:37.976831 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:37.978654 13360 raft_consensus.cc:2410] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260202 15:08:37.978675 12723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1356) took 3992 ms (client timeout 2631 ms). Trace:
I20260202 15:08:37.979810 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.972923 12797 raft_consensus.cc:2393] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:37.956432 14681 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
W20260202 15:08:38.027576 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:37.975327 14724 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:37.969410 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1445) took 2189 ms. Trace:
I20260202 15:08:38.076790 12650 rpcz_store.cc:276] 0202 15:08:35.779604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.834895 (+2055291us) service_pool.cc:225] Handling call
0202 15:08:37.885509 (+ 50614us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:37.885714 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:37.885750 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:37.885807 (+ 57us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:37.885861 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:37.885934 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:37.969304 (+ 83370us) spinlock_profiling.cc:243] Waited 50.3 ms on lock 0x7b540024ed50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.969373 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115768704}
I20260202 15:08:38.035758 12718 raft_consensus.cc:2410] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:37.970474 12614 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:38.035765 12795 raft_consensus.cc:2368] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:37.973353 14616 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:37.974664 14656 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:37.977291 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:38.043452 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:38.087011 12644 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:38.068086 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:38.088322 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:37.977844 14740 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.089742 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.070621 14676 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:38.055660 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.044831 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.031898 14701 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.078984 12798 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:08:38.078913 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 108492us
W20260202 15:08:38.079396 12723 rpcz_store.cc:269] 0202 15:08:33.985918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:33.986070 (+ 152us) service_pool.cc:225] Handling call
0202 15:08:37.978534 (+3992464us) spinlock_profiling.cc:243] Waited 3.74 s on lock 0x7b540033bb54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:37.978621 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8591202560}
I20260202 15:08:38.075062 12724 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:38.080410 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.082085 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:38.083647 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.098865 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:38.089617 12688 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:38.098902 12799 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
W20260202 15:08:38.100363 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 300430us
I20260202 15:08:38.100750 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1743) took 1231 ms. Trace:
I20260202 15:08:38.067188 13359 rpcz_store.cc:276] 0202 15:08:35.870092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.841134 (+1971042us) service_pool.cc:225] Handling call
0202 15:08:37.959183 (+118049us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.101358 12765 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:38.101800 12796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:38.102135 12765 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:38.087791 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 277393us
I20260202 15:08:38.102859 14688 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:38.103103 12764 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.095124 14502 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:38.104086 14502 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:38.087728 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 215055us
I20260202 15:08:38.104223 14619 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:38.103752 14701 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.105162 12763 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:38.105104 14619 consensus_queue.cc:1048] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:08:38.106602 14619 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:38.106936 14619 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:38.107287 14619 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:37.962555 14660 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.090443 13362 raft_consensus.cc:2376] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3. Re-sending same reply.
I20260202 15:08:38.068581 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.099382 12614 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:38.103713 14150 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:38.084179 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:38.102269 12796 raft_consensus.cc:606] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:38.103178 14688 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:38.103698 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 715) took 3300 ms. Trace:
I20260202 15:08:38.104467 14502 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:38.070428 14176 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:38.068277 14715 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:38.084681 14691 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:38.084587 14616 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.109061 12765 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:38.134944 14688 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.037726 14613 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.110080 12764 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:38.161921 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 51913.6us
I20260202 15:08:38.115976 14354 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:38.101055 12799 rpcz_store.cc:276] 0202 15:08:36.869063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.100136 (+1231073us) service_pool.cc:225] Handling call
0202 15:08:38.100366 (+ 230us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.100510 (+ 144us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.100530 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.100566 (+ 36us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:38.100589 (+ 23us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.100633 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.100710 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:38.060726 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 88426.5us
I20260202 15:08:38.109972 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1413) took 3725 ms. Trace:
I20260202 15:08:38.110154 14724 raft_consensus.cc:697] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:38.112311 12614 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:38.086800 14656 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:38.083922 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.042497 14636 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.163755 14613 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.070873 14535 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:38.111729 14494 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:38.218813 14722 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.258859 14722 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:38.229031 14752 log.cc:927] Time spent T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.055s user 0.002s sys 0.000s
I20260202 15:08:38.255414 14535 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.261245 14604 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:38.225729 14751 log.cc:927] Time spent T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260202 15:08:38.263391 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.263185 14724 consensus_queue.cc:237] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.264006 12719 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:38.170020 14354 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.264864 14354 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:38.128607 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.258862 12648 raft_consensus.cc:1240] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:38.116127 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.267351 12725 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:38.090054 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1755) took 1198 ms. Trace:
I20260202 15:08:38.268363 12797 rpcz_store.cc:276] 0202 15:08:36.891743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.869234 (+977491us) service_pool.cc:225] Handling call
0202 15:08:38.087898 (+218664us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.269559 12797 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:38.270081 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1751) took 1384 ms. Trace:
I20260202 15:08:38.258512 14636 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:38.240509 14649 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.272038 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 539) took 3641 ms. Trace:
I20260202 15:08:38.272667 14649 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.098533 14676 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.273644 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1363) took 3447 ms. Trace:
I20260202 15:08:38.273984 12719 rpcz_store.cc:276] 0202 15:08:34.826644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.264613 (+3437969us) service_pool.cc:225] Handling call
0202 15:08:38.264896 (+ 283us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.273363 (+ 8467us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.273393 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.273428 (+ 35us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:38.273460 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.273512 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.273603 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.097429 14683 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:38.095151 14740 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:38.274863 14683 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:38.275243 14683 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.095863 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:38.126075 14674 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:38.111835 14400 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:38.278867 14674 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.279237 14674 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:38.148440 12765 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:38.281005 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 132633us
I20260202 15:08:38.281603 12765 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.282079 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:38.150902 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1747) took 1280 ms. Trace:
I20260202 15:08:38.155776 14150 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260202 15:08:38.139092 14750 log.cc:927] Time spent T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260202 15:08:38.282861 14150 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.283053 13328 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:38.283390 14150 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.283776 14601 raft_consensus.cc:2804] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:38.284154 14601 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:38.284454 14601 raft_consensus.cc:3060] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.135661 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:38.097643 12642 raft_consensus.cc:2393] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:38.168365 12800 rpcz_store.cc:276] 0202 15:08:34.384322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:36.708661 (+2324339us) service_pool.cc:225] Handling call
0202 15:08:36.768710 (+ 60049us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:36.772444 (+ 3734us) raft_consensus.cc:1534] Early marking committed up to index 2
0202 15:08:37.730580 (+958136us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b18000b02c0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4ae335 00007f24f0f59317 00007f24ed569627 00007f24ed578d3f 00007f24ed578cb8
0202 15:08:37.743137 (+ 12557us) log.cc:844] Serialized 14 byte log entry
0202 15:08:38.076755 (+333618us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b18000b02c0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea42646a 00007f24f3aac5c9 00007f24ea2e6160 00007f24ea4ae3c4 00007f24ea4ae335 00007f24ed556ce0 00007f24ed569396 00007f24ed579f07 00007f24ed579e88
0202 15:08:38.085195 (+ 8440us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:38.085520 (+ 325us) log.cc:844] Serialized 26 byte log entry
0202 15:08:38.085621 (+ 101us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:38.089220 (+ 3599us) log.cc:844] Serialized 14 byte log entry
0202 15:08:38.109712 (+ 20492us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.109787 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:38.109820 (+ 33us) raft_consensus.cc:1685] finished
0202 15:08:38.109833 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.109925 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45913,"prepare.queue_time_us":1006529,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":1055,"raft.queue_time_us":302820,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":43417,"spinlock_wait_cycles":6005248,"thread_start_us":54922,"threads_started":2,"wal-append.queue_time_us":50184}
I20260202 15:08:38.041460 14725 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.147475 14673 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:38.142576 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 537) took 3616 ms. Trace:
I20260202 15:08:38.302760 14569 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:38.303217 14569 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:38.302814 13357 rpcz_store.cc:276] 0202 15:08:34.526616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.036214 (+3509598us) service_pool.cc:225] Handling call
0202 15:08:38.090303 (+ 54089us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.115276 (+ 24973us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.126895 (+ 11619us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.134583 (+ 7688us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:38.137883 (+ 3300us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.142403 (+ 4520us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.142522 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.303661 14569 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.304175 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:38.304673 13357 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:38.305500 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:38.306051 13357 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:38.148710 14715 raft_consensus.cc:697] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:38.307305 14715 consensus_queue.cc:237] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.308740 14535 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.308888 14740 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:38.187157 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 74893.3us
I20260202 15:08:38.151711 14691 raft_consensus.cc:697] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:38.311234 14691 consensus_queue.cc:237] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.190824 14306 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:38.168140 14468 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:38.317456 14468 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:38.317799 14468 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.318579 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1744) took 1449 ms. Trace:
I20260202 15:08:38.318794 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:38.042275 14591 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.192225 14634 raft_consensus.cc:3055] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:38.113474 14476 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.244279 14701 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.241447 14176 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:38.323882 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1752) took 1438 ms. Trace:
I20260202 15:08:38.162984 12764 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.326663 12800 rpcz_store.cc:276] 0202 15:08:36.885595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.286095 (+1400500us) service_pool.cc:225] Handling call
0202 15:08:38.323391 (+ 37296us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.323546 (+ 155us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.323575 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.323624 (+ 49us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:38.323656 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.323717 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.323824 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:38.327108 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 164207us
I20260202 15:08:38.270370 12797 rpcz_store.cc:276] 0202 15:08:36.885310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.268998 (+1383688us) service_pool.cc:225] Handling call
0202 15:08:38.269260 (+ 262us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.269827 (+ 567us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.270023 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.327739 12764 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.109647 12763 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:38.286545 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.118366 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1757) took 1209 ms. Trace:
I20260202 15:08:38.328222 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:38.329722 12795 rpcz_store.cc:276] 0202 15:08:36.909161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.083810 (+1174649us) service_pool.cc:225] Handling call
0202 15:08:38.118299 (+ 34489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.133942 13359 rpcz_store.cc:276] 0202 15:08:34.802867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.102222 (+3299355us) service_pool.cc:225] Handling call
0202 15:08:38.102761 (+ 539us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.103094 (+ 333us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.103162 (+ 68us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.103246 (+ 84us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:38.103308 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.103425 (+ 117us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.103616 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.272354 13360 rpcz_store.cc:276] 0202 15:08:34.630865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.081554 (+3450689us) service_pool.cc:225] Handling call
0202 15:08:38.271115 (+189561us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.271271 (+ 156us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.271291 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.271320 (+ 29us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:38.271343 (+ 23us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.271384 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.271908 (+ 524us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54005eddd0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:38.271994 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402010368}
I20260202 15:08:38.282711 12794 rpcz_store.cc:276] 0202 15:08:36.870411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.608603 (+738192us) service_pool.cc:225] Handling call
0202 15:08:38.150642 (+542039us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b54000cc454. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:38.150835 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83420544}
I20260202 15:08:38.297304 14400 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260202 15:08:38.289192 13361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 518) took 1205 ms (client timeout 1578 ms). Trace:
W20260202 15:08:38.336185 13361 rpcz_store.cc:269] 0202 15:08:37.083621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.604870 (+521249us) service_pool.cc:225] Handling call
0202 15:08:38.289038 (+684168us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b5400511654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:38.289145 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467552000}
I20260202 15:08:38.282575 12765 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.292459 12689 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:38.338181 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 45777.1us
I20260202 15:08:38.338902 12689 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.339995 12689 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.340716 12724 raft_consensus.cc:1240] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:38.301239 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:38.282563 12719 raft_consensus.cc:2376] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3. Re-sending same reply.
I20260202 15:08:38.211704 12792 raft_consensus.cc:1275] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260202 15:08:38.313268 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.314198 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.315323 12546 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39704 (request call id 239) took 7888 ms. Trace:
I20260202 15:08:38.349579 12800 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260202 15:08:38.350219 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1753) took 1464 ms. Trace:
I20260202 15:08:38.350520 12800 rpcz_store.cc:276] 0202 15:08:36.885867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.327473 (+1441606us) service_pool.cc:225] Handling call
0202 15:08:38.346496 (+ 19023us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.349990 (+ 3494us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.350166 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.315025 14306 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.342521 14693 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:38.308552 12544 catalog_manager.cc:5654] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:38.320459 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.325568 14634 raft_consensus.cc:740] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:38.328205 14511 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:38.272830 13355 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:38.284643 13328 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.363626 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.312879 14494 raft_consensus.cc:697] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:38.256392 14711 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.345793 13363 raft_consensus.cc:2393] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:38.347976 12794 raft_consensus.cc:1240] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260202 15:08:38.194705 12717 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:38.352083 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1361) took 3826 ms. Trace:
I20260202 15:08:38.360199 12544 catalog_manager.cc:5654] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:38.348505 14740 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:38.371485 12724 rpcz_store.cc:276] 0202 15:08:34.525391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.108219 (+3582828us) service_pool.cc:225] Handling call
0202 15:08:38.108464 (+ 245us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.341086 (+232622us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.351880 (+ 10794us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b540033b154. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:38.352025 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":533677312}
I20260202 15:08:38.372198 14740 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:38.372277 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 496) took 2558 ms. Trace:
I20260202 15:08:38.372484 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.372658 13355 rpcz_store.cc:276] 0202 15:08:35.813565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.262330 (+2448765us) service_pool.cc:225] Handling call
0202 15:08:38.262678 (+ 348us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.369288 (+106610us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.372227 (+ 2939us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.373373 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.373484 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.373961 12724 raft_consensus.cc:2393] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:38.374257 14701 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:38.338256 14559 consensus_queue.cc:1059] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260202 15:08:38.358871 13358 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:38.375187 14559 raft_consensus.cc:3055] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:38.337510 14660 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.375567 14559 raft_consensus.cc:740] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:38.375618 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 723) took 1008 ms. Trace:
I20260202 15:08:38.376418 14559 consensus_queue.cc:260] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.377068 14660 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.377645 14559 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.349460 12546 rpcz_store.cc:276] 0202 15:08:30.426997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:30.435902 (+ 8905us) service_pool.cc:225] Handling call
0202 15:08:38.315272 (+7879370us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:08:37.060452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:08:37.385754 (+325302us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:08:37.405140 (+ 19386us) write_op.cc:435] Acquired schema lock
0202 15:08:37.408275 (+ 3135us) tablet.cc:660] Decoding operations
0202 15:08:37.587043 (+178768us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:08:37.591102 (+ 4059us) write_op.cc:641] Partition lock acquired for write op
0202 15:08:37.596330 (+ 5228us) tablet.cc:683] Acquiring locks for 3 operations
0202 15:08:37.623273 (+ 26943us) tablet.cc:699] Row locks acquired
0202 15:08:37.623288 (+ 15us) write_op.cc:260] PREPARE: finished
0202 15:08:37.623412 (+ 124us) write_op.cc:270] Start()
0202 15:08:37.623508 (+ 96us) write_op.cc:276] Timestamp: P: 1770044917623372 usec, L: 0
0202 15:08:37.623531 (+ 23us) op_driver.cc:348] REPLICATION: starting
0202 15:08:37.649672 (+ 26141us) log.cc:844] Serialized 1302 byte log entry
0202 15:08:37.777392 (+127720us) op_driver.cc:464] REPLICATION: finished
0202 15:08:37.956492 (+179100us) write_op.cc:301] APPLY: starting
0202 15:08:37.956594 (+ 102us) tablet.cc:1365] starting BulkCheckPresence
0202 15:08:37.956703 (+ 109us) tablet.cc:1368] finished BulkCheckPresence
0202 15:08:37.979270 (+ 22567us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:08:38.036106 (+ 56836us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:08:38.067988 (+ 31882us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0202 15:08:38.068028 (+ 40us) write_op.cc:312] APPLY: finished
0202 15:08:38.070379 (+ 2351us) log.cc:844] Serialized 35 byte log entry
0202 15:08:38.083439 (+ 13060us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:38.087376 (+ 3937us) write_op.cc:454] Released schema lock
0202 15:08:38.090105 (+ 2729us) write_op.cc:341] FINISH: Updating metrics
0202 15:08:38.100208 (+ 10103us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:38.100231 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1371033,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":175794,"apply.run_cpu_time_us":2761,"apply.run_wall_time_us":143731,"num_ops":3,"prepare.queue_time_us":821876,"prepare.run_cpu_time_us":8928,"prepare.run_wall_time_us":707545,"raft.queue_time_us":40649,"raft.run_cpu_time_us":15462,"raft.run_wall_time_us":121303,"replication_time_us":153726,"thread_start_us":901149,"threads_started":4,"wal-append.queue_time_us":69212}]]}
I20260202 15:08:38.377930 14494 consensus_queue.cc:237] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.378794 14535 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:38.353996 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.380155 12797 raft_consensus.cc:1240] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:38.380249 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.381829 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1366) took 1536 ms. Trace:
I20260202 15:08:38.382135 12797 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:38.375203 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.383262 14616 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.325469 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:38.383991 14711 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.331703 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:38.381980 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.385058 13361 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:38.385551 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:38.386533 13357 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:38.389391 13326 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.391081 14699 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:38.374593 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.341414 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1427) took 1035 ms. Trace:
W20260202 15:08:38.367554 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 83094.8us
I20260202 15:08:38.392905 12722 raft_consensus.cc:2410] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:38.393882 12797 raft_consensus.cc:1240] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260202 15:08:38.394171 13328 leader_election.cc:336] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.973s (SENT)
I20260202 15:08:38.394363 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1377) took 1394 ms. Trace:
I20260202 15:08:38.394554 13328 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:38.394644 12722 rpcz_store.cc:276] 0202 15:08:36.999976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.805659 (+805683us) service_pool.cc:225] Handling call
0202 15:08:38.394186 (+588527us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b5400327dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:38.394319 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1253510016}
I20260202 15:08:38.395758 12722 raft_consensus.cc:1240] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.395834 14699 raft_consensus.cc:2749] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:38.396036 13328 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.396291 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1652) took 1546 ms. Trace:
I20260202 15:08:38.324712 14674 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.318948 12798 rpcz_store.cc:276] 0202 15:08:36.869364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.105537 (+1236173us) service_pool.cc:225] Handling call
0202 15:08:38.105829 (+ 292us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:38.106169 (+ 340us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:38.106417 (+ 248us) log.cc:844] Serialized 14 byte log entry
0202 15:08:38.183245 (+ 76828us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:38.193929 (+ 10684us) log.cc:844] Serialized 26 byte log entry
0202 15:08:38.194116 (+ 187us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:38.194166 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.194231 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:38.318383 (+124152us) raft_consensus.cc:1685] finished
0202 15:08:38.318401 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.318529 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1920,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":56,"raft.queue_time_us":2493,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":7786,"thread_start_us":76158,"threads_started":1,"wal-append.queue_time_us":106581}
I20260202 15:08:38.397833 14616 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:38.392711 12791 rpcz_store.cc:276] 0202 15:08:37.300532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.586198 (+285666us) service_pool.cc:225] Handling call
0202 15:08:37.728247 (+142049us) spinlock_profiling.cc:243] Waited 6.79 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:38.125387 (+397140us) spinlock_profiling.cc:243] Waited 5.73 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea4740dc 00007f24ea473c9b 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:38.335609 (+210222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28791296}
I20260202 15:08:38.383447 12726 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:38.401011 14569 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.384339 12721 raft_consensus.cc:2393] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:08:38.364389 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 267481us
I20260202 15:08:38.362179 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.387626 14591 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:38.402938 12649 raft_consensus.cc:1240] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260202 15:08:38.387388 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.403904 12646 raft_consensus.cc:2368] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5.
I20260202 15:08:38.388998 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.405025 12688 leader_election.cc:400] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5.
I20260202 15:08:38.405340 12688 leader_election.cc:403] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260202 15:08:38.402592 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1662) took 1404 ms. Trace:
I20260202 15:08:38.353889 12689 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:38.406934 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 53098.1us
I20260202 15:08:38.407146 14711 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.407634 12689 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:38.409547 14591 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:38.408339 13361 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:38.376164 13358 rpcz_store.cc:276] 0202 15:08:37.367633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.032229 (+664596us) service_pool.cc:225] Handling call
0202 15:08:38.301120 (+268891us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea47412e 000000000035a3c9 00007f24f249e560 00007f24f2566dd4 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:38.375560 (+ 74440us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6513536}
I20260202 15:08:38.411849 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.410881 14681 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:38.412443 12614 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:38.412858 14681 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.406309 14616 consensus_queue.cc:1059] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260202 15:08:38.413386 14681 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.414108 14657 raft_consensus.cc:3055] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:38.414637 14657 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:38.415848 14150 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.415693 14657 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.417356 14657 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:38.398495 14674 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.410073 14591 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:38.417356 12793 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:38.406194 12721 rpcz_store.cc:276] 0202 15:08:36.998524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.075907 (+1077383us) service_pool.cc:225] Handling call
0202 15:08:38.402396 (+326489us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54000172d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:38.402526 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":679227136}
I20260202 15:08:38.418429 14674 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:38.419998 12721 raft_consensus.cc:1240] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:38.419947 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1421) took 1291 ms. Trace:
I20260202 15:08:38.418895 14591 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:38.420900 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.420575 12793 rpcz_store.cc:276] 0202 15:08:37.128282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.072252 (+943970us) service_pool.cc:225] Handling call
0202 15:08:38.419735 (+347483us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b5400372dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:38.419892 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":318391680}
W20260202 15:08:38.421455 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.382150 12717 rpcz_store.cc:276] 0202 15:08:36.844963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.381113 (+1536150us) service_pool.cc:225] Handling call
0202 15:08:38.381380 (+ 267us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.381529 (+ 149us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.381552 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.381595 (+ 43us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:38.381625 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.381677 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.381782 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.390373 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:38.364923 14636 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:38.365629 14604 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:38.355943 14306 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:38.420680 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1368) took 1532 ms. Trace:
I20260202 15:08:38.408965 12689 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.359495 14258 raft_consensus.cc:2804] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:38.356544 14400 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:38.359593 13354 raft_consensus.cc:1240] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:38.359220 14693 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.422483 13361 raft_consensus.cc:1240] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:38.426441 12721 rpcz_store.cc:276] 0202 15:08:36.888472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.419530 (+1531058us) service_pool.cc:225] Handling call
0202 15:08:38.419845 (+ 315us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.420464 (+ 619us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.420625 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.435087 13328 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.447839 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 501) took 2626 ms. Trace:
I20260202 15:08:38.448597 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.448788 13328 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:38.438189 12544 catalog_manager.cc:5654] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:38.450073 12689 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:38.374390 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:38.427122 12724 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:38.455314 13328 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:38.451252 12689 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.444321 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:38.396641 12722 rpcz_store.cc:276] 0202 15:08:36.849738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.395301 (+1545563us) service_pool.cc:225] Handling call
0202 15:08:38.395617 (+ 316us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.396082 (+ 465us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.396248 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.418397 14150 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.448179 13361 rpcz_store.cc:276] 0202 15:08:35.821905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.411773 (+2589868us) service_pool.cc:225] Handling call
0202 15:08:38.416226 (+ 4453us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.447654 (+ 31428us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.447797 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.459537 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:38.460420 12724 raft_consensus.cc:2410] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260202 15:08:38.454922 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.427695 14258 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:38.426047 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.428009 14604 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:38.439863 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:38.427043 14674 raft_consensus.cc:697] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:38.461895 13359 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:38.462394 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:38.434012 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1745) took 1501 ms. Trace:
I20260202 15:08:38.423564 12717 raft_consensus.cc:1275] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260202 15:08:38.457556 14636 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:38.452613 12544 catalog_manager.cc:5654] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 2 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:38.465633 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.441812 14535 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:38.493711 12544 catalog_manager.cc:5654] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 3 to 4, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 4 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:38.494625 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:38.447765 14400 raft_consensus.cc:697] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:38.499542 12722 raft_consensus.cc:2410] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260202 15:08:38.426493 12719 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:38.500152 13328 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.492880 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:38.462486 14258 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:38.462359 14176 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:38.424537 14502 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.496071 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.502306 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:38.496229 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.503446 12689 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:38.462236 14740 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:38.463418 13355 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:38.509531 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1655) took 1618 ms. Trace:
I20260202 15:08:38.524461 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 513) took 2281 ms. Trace:
I20260202 15:08:38.524582 12717 rpcz_store.cc:276] 0202 15:08:36.891294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.422648 (+1531354us) service_pool.cc:225] Handling call
0202 15:08:38.423129 (+ 481us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.492356 (+ 69227us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.508283 (+ 15927us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.524821 13360 rpcz_store.cc:276] 0202 15:08:36.243187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.523718 (+2280531us) service_pool.cc:225] Handling call
0202 15:08:38.523998 (+ 280us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.524167 (+ 169us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.524186 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.524234 (+ 48us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:38.524283 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.524332 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.524418 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.525668 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:38.526171 13360 raft_consensus.cc:1240] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.526244 12717 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:38.528496 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 543) took 2166 ms. Trace:
I20260202 15:08:38.528805 13360 rpcz_store.cc:276] 0202 15:08:36.362254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.525734 (+2163480us) service_pool.cc:225] Handling call
0202 15:08:38.526007 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.526629 (+ 622us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.528449 (+ 1820us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.529873 14649 consensus_queue.cc:1059] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:38.510772 12722 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.535283 14400 consensus_queue.cc:237] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.536315 14688 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.537317 12796 raft_consensus.cc:1240] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:38.538218 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1746) took 1668 ms. Trace:
I20260202 15:08:38.511003 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:38.538875 14564 consensus_meta.cc:287] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.702s user 0.005s sys 0.004s
I20260202 15:08:38.492606 14636 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.539232 12724 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.539309 14564 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.541476 12643 raft_consensus.cc:1240] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.543113 13359 raft_consensus.cc:1240] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:38.543241 14616 consensus_queue.cc:1059] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:38.543666 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 507) took 2681 ms. Trace:
I20260202 15:08:38.544117 14616 raft_consensus.cc:3055] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:38.544445 14616 raft_consensus.cc:740] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:38.459260 14634 consensus_queue.cc:260] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.499603 14535 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.545079 14616 consensus_queue.cc:260] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.518829 14740 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:38.546231 14616 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.518656 14674 consensus_queue.cc:237] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.546794 14502 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.548161 14569 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:38.546622 14740 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:38.518790 13357 raft_consensus.cc:2393] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:38.549821 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.455221 14676 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.518553 14476 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:38.551703 12800 raft_consensus.cc:1240] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:38.539546 14636 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.560039 12722 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:38.530686 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 515) took 2166 ms. Trace:
I20260202 15:08:38.561396 13360 rpcz_store.cc:276] 0202 15:08:36.364099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.530045 (+2165946us) service_pool.cc:225] Handling call
0202 15:08:38.530292 (+ 247us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.530441 (+ 149us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.530465 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.530499 (+ 34us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:38.530523 (+ 24us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.530569 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.530648 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.556031 12724 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:38.538471 12796 rpcz_store.cc:276] 0202 15:08:36.870008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.143867 (+1273859us) service_pool.cc:225] Handling call
0202 15:08:38.164135 (+ 20268us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.537608 (+373473us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.538047 (+ 439us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b54001f7cd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:38.538171 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":857860480}
I20260202 15:08:38.444931 12615 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:38.530719 14649 raft_consensus.cc:3055] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:38.519517 14468 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.605931 12719 raft_consensus.cc:1240] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:38.606410 14722 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:38.531457 14613 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.608530 14656 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.534686 14725 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:38.515585 13356 consensus_meta.cc:287] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.552s user 0.001s sys 0.007s
I20260202 15:08:38.561300 14564 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.551937 14701 raft_consensus.cc:2749] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:38.523697 13355 raft_consensus.cc:1240] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:38.534909 14306 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.545535 14535 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.612728 14693 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:38.614888 13358 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:38.616621 14676 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:38.539268 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 538) took 4011 ms. Trace:
I20260202 15:08:38.531479 14619 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.617673 14711 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5.
I20260202 15:08:38.543253 14354 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.514590 14481 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:38.510176 14176 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:38.530426 14591 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:38.600896 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 156052us
I20260202 15:08:38.600857 12722 raft_consensus.cc:1240] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.492357 12792 rpcz_store.cc:276] 0202 15:08:36.869637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.139351 (+1269714us) service_pool.cc:225] Handling call
0202 15:08:38.174833 (+ 35482us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.353579 (+178746us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.370550 (+ 16971us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.629866 12799 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.659438 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 544) took 2291 ms. Trace:
I20260202 15:08:38.637064 12723 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:38.662257 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 273) took 4220 ms. Trace:
I20260202 15:08:38.662523 12723 rpcz_store.cc:276] 0202 15:08:34.442069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.100607 (+3658538us) service_pool.cc:225] Handling call
0202 15:08:38.100835 (+ 228us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.661633 (+560798us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.662086 (+ 453us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54000177d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:38.662215 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1232714496}
I20260202 15:08:38.663985 12723 raft_consensus.cc:1240] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260202 15:08:38.533923 14601 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.630746 14258 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.639014 14725 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:38.631356 13356 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:38.666561 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 547) took 1076 ms. Trace:
I20260202 15:08:38.666929 13356 rpcz_store.cc:276] 0202 15:08:37.590415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:37.887419 (+297004us) service_pool.cc:225] Handling call
0202 15:08:38.666501 (+779082us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.667348 14601 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:38.667580 14258 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.668047 14699 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:38.668339 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.669291 13363 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260202 15:08:38.669366 12726 raft_consensus.cc:1240] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:38.639567 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 541) took 2281 ms. Trace:
I20260202 15:08:38.640925 14306 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.636521 13354 rpcz_store.cc:276] 0202 15:08:34.527828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.080999 (+3553171us) service_pool.cc:225] Handling call
0202 15:08:38.137629 (+ 56630us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.447438 (+309809us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.539061 (+ 91623us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54004dc6d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:38.539221 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":457403648}
I20260202 15:08:38.644459 12642 raft_consensus.cc:2368] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6.
W20260202 15:08:38.644838 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.645205 14634 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.648343 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.621726 14722 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.619601 14649 raft_consensus.cc:740] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:38.518392 14681 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.659118 12615 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:38.543995 13359 rpcz_store.cc:276] 0202 15:08:35.862679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.510145 (+2647466us) service_pool.cc:225] Handling call
0202 15:08:38.542945 (+ 32800us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.543457 (+ 512us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.543616 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.659895 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1373) took 1726 ms. Trace:
I20260202 15:08:38.670572 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 719) took 2839 ms. Trace:
I20260202 15:08:38.688184 13363 rpcz_store.cc:276] 0202 15:08:35.830776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.502008 (+2671232us) service_pool.cc:225] Handling call
0202 15:08:38.669074 (+167066us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:38.669640 (+ 566us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.670417 (+ 777us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54006ade50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:38.670524 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":334443904}
I20260202 15:08:38.671638 12723 raft_consensus.cc:1240] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.671932 12795 raft_consensus.cc:1240] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:38.649264 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1372) took 1726 ms. Trace:
I20260202 15:08:38.691305 12723 raft_consensus.cc:1240] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.672421 12721 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:38.637535 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.681478 12724 raft_consensus.cc:1240] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.655754 14481 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:38.672796 12614 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:38.672200 13360 rpcz_store.cc:276] 0202 15:08:36.366793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.619383 (+2252590us) service_pool.cc:225] Handling call
0202 15:08:38.658080 (+ 38697us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.658242 (+ 162us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.658264 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.658299 (+ 35us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:38.658323 (+ 24us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.658367 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.658451 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.672942 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1654) took 1784 ms. Trace:
I20260202 15:08:38.657299 14176 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.694011 12726 rpcz_store.cc:276] 0202 15:08:36.888037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.401427 (+1513390us) service_pool.cc:225] Handling call
0202 15:08:38.401675 (+ 248us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.669698 (+268023us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.672773 (+ 3075us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54003acfd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:38.672900 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":615285120}
I20260202 15:08:38.672989 12725 raft_consensus.cc:1240] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260202 15:08:38.675629 13357 raft_consensus.cc:2393] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:38.701893 13326 leader_election.cc:400] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6.
I20260202 15:08:38.702242 13326 leader_election.cc:403] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260202 15:08:38.674167 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1748) took 1803 ms. Trace:
I20260202 15:08:38.675344 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.703565 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1435) took 1119 ms. Trace:
I20260202 15:08:38.703169 12799 rpcz_store.cc:276] 0202 15:08:36.870674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.171107 (+1300433us) service_pool.cc:225] Handling call
0202 15:08:38.171440 (+ 333us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.660995 (+489555us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.673933 (+ 12938us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b540036e054. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:38.674107 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1053867264}
W20260202 15:08:38.687561 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 577945us
W20260202 15:08:38.699301 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.684787 13358 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:38.705821 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1365) took 1861 ms. Trace:
I20260202 15:08:38.710630 12725 rpcz_store.cc:276] 0202 15:08:36.844635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.267866 (+1423231us) service_pool.cc:225] Handling call
0202 15:08:38.268149 (+ 283us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:38.695821 (+427672us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.705551 (+ 9730us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b540033b654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:38.705757 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":930573696}
I20260202 15:08:38.684638 12717 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:38.684219 14722 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.704051 12795 rpcz_store.cc:276] 0202 15:08:37.584167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.330428 (+746261us) service_pool.cc:225] Handling call
0202 15:08:38.353191 (+ 22763us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.689900 (+336709us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.703347 (+ 13447us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54001f8bd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:38.703511 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":732644096}
I20260202 15:08:38.708263 12723 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260202 15:08:38.670166 13355 rpcz_store.cc:276] 0202 15:08:36.358360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.523235 (+2164875us) service_pool.cc:225] Handling call
0202 15:08:38.523517 (+ 282us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.634399 (+110882us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.639510 (+ 5111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.712893 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.723737 12642 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:38.717453 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.724200 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.701015 14481 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.692616 14306 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:38.718997 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.726873 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:38.719312 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1375) took 1734 ms. Trace:
I20260202 15:08:38.732765 12724 rpcz_store.cc:276] 0202 15:08:36.985076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.620383 (+1635307us) service_pool.cc:225] Handling call
0202 15:08:38.620672 (+ 289us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.692386 (+ 71714us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.719134 (+ 26748us) spinlock_profiling.cc:243] Waited 60.6 ms on lock 0x7b5400144e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:38.719265 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139429120}
I20260202 15:08:38.733353 12725 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:38.708066 14691 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:38.713884 14564 consensus_queue.cc:1059] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260202 15:08:38.722270 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.737346 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:38.738291 12799 raft_consensus.cc:2393] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:38.693434 12722 rpcz_store.cc:276] 0202 15:08:36.933703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.600490 (+1666787us) service_pool.cc:225] Handling call
0202 15:08:38.600718 (+ 228us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.659553 (+ 58835us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.659845 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.739161 13360 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:38.729445 14619 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:38.720163 14150 consensus_queue.cc:1059] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260202 15:08:38.730101 14468 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.730595 14354 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:38.733857 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.687009 14683 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.739960 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 528) took 1097 ms. Trace:
I20260202 15:08:38.740721 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.734426 12643 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:38.720422 14688 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:38.756529 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:38.756948 13363 raft_consensus.cc:1240] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.737149 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1379) took 1729 ms. Trace:
I20260202 15:08:38.758001 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:38.759714 12796 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.692812 12719 rpcz_store.cc:276] 0202 15:08:36.922925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.544768 (+1621843us) service_pool.cc:225] Handling call
0202 15:08:38.605664 (+ 60896us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.631812 (+ 26148us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.649076 (+ 17264us) spinlock_profiling.cc:243] Waited 56 ms on lock 0x7b5400146750. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:38.649210 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":128755584}
I20260202 15:08:38.757593 14656 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.752633 13360 rpcz_store.cc:276] 0202 15:08:37.642828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.694068 (+1051240us) service_pool.cc:225] Handling call
0202 15:08:38.735474 (+ 41406us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.739664 (+ 4190us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.739900 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.756006 14468 raft_consensus.cc:3055] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:38.753906 14306 consensus_queue.cc:1059] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.010s
I20260202 15:08:38.759004 14613 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:38.726436 14649 consensus_queue.cc:260] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:38.786707 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 458990us
I20260202 15:08:38.791868 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 721) took 1424 ms. Trace:
I20260202 15:08:38.793143 13363 rpcz_store.cc:276] 0202 15:08:37.367085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.689170 (+1322085us) service_pool.cc:225] Handling call
0202 15:08:38.755355 (+ 66185us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.757542 (+ 2187us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.791660 (+ 34118us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b5400483350. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:38.791808 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97225600}
I20260202 15:08:38.784483 14468 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:38.793385 12796 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
W20260202 15:08:38.799574 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.803181 12764 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.804662 12764 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:38.758518 14602 consensus_queue.cc:1059] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260202 15:08:38.807206 14681 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.807611 14649 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.808012 14468 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.809128 14468 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.762377 12717 rpcz_store.cc:276] 0202 15:08:37.007316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.629829 (+1622513us) service_pool.cc:225] Handling call
0202 15:08:38.657470 (+ 27641us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.719201 (+ 61731us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.734745 (+ 15544us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.811252 14150 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:38.811705 14150 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:38.812039 14150 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:38.813920 14481 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:38.811347 13363 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEARNER]: Advancing to term 6
W20260202 15:08:38.818234 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.819576 13360 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:38.821661 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 556) took 1167 ms. Trace:
I20260202 15:08:38.823431 13360 rpcz_store.cc:276] 0202 15:08:37.654016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.767538 (+1113522us) service_pool.cc:225] Handling call
0202 15:08:38.819375 (+ 51837us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.821405 (+ 2030us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.821603 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:38.830510 12764 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.837785 12764 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:38.819950 14715 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.016s sys 0.038s
I20260202 15:08:38.794267 14591 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:38.816362 14601 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:38.841424 13360 raft_consensus.cc:1240] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:38.835523 14688 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260202 15:08:38.842455 14715 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s user 0.016s sys 0.038s
W20260202 15:08:38.850142 12765 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.013s sys 0.006s
W20260202 15:08:38.855168 12765 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s user 0.014s sys 0.006s
I20260202 15:08:38.843677 14502 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:38.846323 13357 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:38.846751 14601 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:38.848627 14688 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:38.856011 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 573467us
W20260202 15:08:38.957886 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:38.958993 12765 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:38.961571 12765 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:38.955327 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 451722us
W20260202 15:08:38.955873 12718 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.004s sys 0.024s
W20260202 15:08:38.971311 12718 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.690s user 0.004s sys 0.024s
I20260202 15:08:38.973210 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 271) took 4597 ms. Trace:
W20260202 15:08:38.956161 12720 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.001s sys 0.014s
W20260202 15:08:38.975116 12720 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s user 0.001s sys 0.014s
I20260202 15:08:38.973680 12718 rpcz_store.cc:276] 0202 15:08:34.375445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.090597 (+3715152us) service_pool.cc:225] Handling call
0202 15:08:38.113181 (+ 22584us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:38.113910 (+ 729us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.113936 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:38.135678 (+ 21742us) log.cc:844] Serialized 52 byte log entry
0202 15:08:38.248273 (+112595us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:38.971936 (+723663us) log.cc:844] Serialized 14 byte log entry
0202 15:08:38.972502 (+ 566us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.972667 (+ 165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:38.972780 (+ 113us) raft_consensus.cc:1685] finished
0202 15:08:38.972804 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:38.973070 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":18994,"thread_start_us":782042,"threads_started":2,"wal-append.queue_time_us":280338}
W20260202 15:08:38.989790 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:38.991169 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:39.012122 13329 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.012s sys 0.040s
W20260202 15:08:39.012576 13329 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.012s sys 0.040s
I20260202 15:08:39.012392 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 720) took 1645 ms. Trace:
I20260202 15:08:39.013119 13354 rpcz_store.cc:276] 0202 15:08:37.366760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.683237 (+1316477us) service_pool.cc:225] Handling call
0202 15:08:38.834194 (+150957us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea47412e 000000000035a3c9 00007f24f249e560 00007f24f2566dd4 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:38.839516 (+ 5322us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.846270 (+ 6754us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.847581 (+ 1311us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:38.853894 (+ 6313us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:38.855259 (+ 1365us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.865594 (+ 10335us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:39.012085 (+146491us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10950400}
I20260202 15:08:39.019614 13362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 540) took 4120 ms. Trace:
I20260202 15:08:39.021018 13354 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:39.022141 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 558) took 1367 ms. Trace:
I20260202 15:08:39.022686 13357 rpcz_store.cc:276] 0202 15:08:37.654552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.785396 (+1130844us) service_pool.cc:225] Handling call
0202 15:08:38.836894 (+ 51498us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:39.021231 (+184337us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:39.021895 (+ 664us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:39.020197 13362 rpcz_store.cc:276] 0202 15:08:34.898847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.108610 (+3209763us) service_pool.cc:225] Handling call
0202 15:08:38.137484 (+ 28874us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:38.353579 (+216095us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.353604 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:38.354019 (+ 415us) log.cc:844] Serialized 26 byte log entry
0202 15:08:38.836831 (+482812us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:38.836928 (+ 97us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:38.837024 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:39.018933 (+181909us) raft_consensus.cc:1685] finished
0202 15:08:39.018991 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:39.019322 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5125,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":4258,"thread_start_us":644230,"threads_started":2,"wal-append.queue_time_us":514177}
I20260202 15:08:39.024356 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.869575 14601 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:39.041600 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 80285.9us
I20260202 15:08:39.041960 14494 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.056128 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:39.059031 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.034531 14481 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.062664 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:39.071970 13358 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:39.076795 12792 raft_consensus.cc:1240] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:39.050794 12765 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.125521 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:39.126945 14660 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.007s sys 0.009s
I20260202 15:08:39.143150 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.134474 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.925611 14724 raft_consensus.cc:3055] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:38.888427 14688 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.138649 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:38.868608 14683 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:39.162273 14660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.008s sys 0.010s
I20260202 15:08:39.160223 13358 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.170245 14699 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6.
I20260202 15:08:39.170665 14604 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:39.170993 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:39.172924 14724 raft_consensus.cc:740] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.173421 14619 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:38.925390 14722 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.182022 12645 raft_consensus.cc:1240] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260202 15:08:39.183252 14722 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:39.184425 14722 raft_consensus.cc:3055] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:39.184896 14722 raft_consensus.cc:740] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:38.897249 14636 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.182284 14724 consensus_queue.cc:260] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.187964 13362 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 2: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:39.188679 14636 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.189023 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.190240 14764 raft_consensus.cc:2804] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:39.193634 14764 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:39.194079 14764 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:39.155304 14494 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:39.179172 14771 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:39.196746 14771 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:38.924515 14740 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:39.187805 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 197653us
I20260202 15:08:39.174893 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:39.211066 12800 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:39.223023 13363 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260202 15:08:39.223912 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 559) took 1373 ms. Trace:
I20260202 15:08:39.224076 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:39.224287 13363 rpcz_store.cc:276] 0202 15:08:37.854783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.810940 (+956157us) service_pool.cc:225] Handling call
0202 15:08:38.811215 (+ 275us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:39.223557 (+412342us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:39.223849 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:39.222416 13362 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:39.227355 14535 consensus_meta.cc:287] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.565s user 0.008s sys 0.008s
I20260202 15:08:39.225282 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.224761 12645 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:39.227587 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:39.229564 13361 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.007s sys 0.036s
I20260202 15:08:39.231432 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:39.246254 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 203778us
W20260202 15:08:39.234434 13328 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.010s sys 0.014s
W20260202 15:08:39.245518 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 122436us
W20260202 15:08:39.247124 13361 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.008s sys 0.037s
W20260202 15:08:39.241232 14400 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.015s sys 0.012s
W20260202 15:08:39.250671 13328 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.745s user 0.011s sys 0.014s
I20260202 15:08:39.252651 13354 raft_consensus.cc:2393] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:39.257351 12795 raft_consensus.cc:1240] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
W20260202 15:08:39.257830 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 478600us
I20260202 15:08:39.259794 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.262379 14724 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.262921 12688 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:39.233803 14764 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.246304 14771 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:39.266332 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 489844us
I20260202 15:08:39.269325 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:39.254498 14400 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s user 0.016s sys 0.012s
I20260202 15:08:39.271615 14740 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:39.277055 14634 consensus_meta.cc:287] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.557s user 0.010s sys 0.000s
I20260202 15:08:39.277457 12615 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:39.277802 14722 consensus_queue.cc:260] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.278898 14715 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:39.279304 12690 leader_election.cc:336] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.893s (SENT)
I20260202 15:08:39.279820 12690 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.279659 14715 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.281229 14258 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:39.281629 14258 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:39.281836 14715 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.282006 14258 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.282264 12645 raft_consensus.cc:1275] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260202 15:08:39.277155 12792 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:39.279320 14722 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:39.284759 14150 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.287101 14150 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:39.287559 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:39.288245 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:39.288609 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:39.289032 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.289978 14601 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.291332 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:39.290621 14602 consensus_queue.cc:1059] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:08:39.307600 14602 consensus_queue.cc:1243] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:08:39.242552 14691 raft_consensus.cc:697] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.308727 12796 raft_consensus.cc:1240] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:39.309549 14691 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.291034 14674 consensus_queue.cc:1059] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260202 15:08:39.311185 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:39.311487 14674 raft_consensus.cc:3055] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:39.311898 14674 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.292361 14619 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:39.293958 12765 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:39.295070 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:39.296952 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.295495 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:39.295821 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.297643 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.297897 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.312958 14674 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.303015 14787 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:39.304276 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.305341 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.305665 12615 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:39.282409 14715 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:39.243464 14535 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.286419 14688 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.298754 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.327957 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.308265 14602 raft_consensus.cc:3055] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:39.329043 12797 raft_consensus.cc:1240] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:39.329128 14602 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.330479 12793 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:39.330454 14602 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.332777 14602 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.257084 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1362) took 4427 ms. Trace:
I20260202 15:08:39.318914 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 505) took 3488 ms. Trace:
I20260202 15:08:39.309573 12689 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.320745 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:39.294564 13357 raft_consensus.cc:2468] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:39.320636 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.326725 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.326896 13360 raft_consensus.cc:1240] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:39.328104 14688 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.328655 14150 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:39.299602 14634 consensus_queue.cc:1048] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:39.337320 14535 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:39.342885 12798 raft_consensus.cc:2393] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:39.343039 14306 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:39.322857 14787 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:39.349376 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1675) took 1042 ms. Trace:
I20260202 15:08:39.345956 12791 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:39.345904 12765 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.324389 14601 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.343866 12689 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.343827 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.358870 14400 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:39.359266 14400 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:39.359587 14649 raft_consensus.cc:3055] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:39.360384 14649 raft_consensus.cc:740] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.354892 14306 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:39.361671 14306 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.361801 12765 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:39.340451 13361 rpcz_store.cc:276] 0202 15:08:35.830537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.461303 (+2630766us) service_pool.cc:225] Handling call
0202 15:08:38.461640 (+ 337us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:38.461922 (+ 282us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.461947 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:38.579432 (+117485us) log.cc:844] Serialized 26 byte log entry
0202 15:08:39.301315 (+721883us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:39.301758 (+ 443us) log.cc:844] Serialized 14 byte log entry
0202 15:08:39.302113 (+ 355us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:39.302221 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:39.313677 (+ 11456us) raft_consensus.cc:1685] finished
0202 15:08:39.313692 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:39.318853 (+ 5161us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":244,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":45353,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":33417,"thread_start_us":618462,"threads_started":1,"wal-append.queue_time_us":704850}
I20260202 15:08:39.363629 14619 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:39.343564 12615 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:39.359688 14400 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:39.356488 14787 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:39.364357 12690 leader_election.cc:336] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.937s (SENT)
I20260202 15:08:39.366868 14564 consensus_queue.cc:1059] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260202 15:08:39.368606 14176 consensus_meta.cc:287] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.659s user 0.015s sys 0.006s
I20260202 15:08:39.348376 14674 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.368332 14649 consensus_queue.cc:260] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.348011 14150 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:39.374500 12690 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.352587 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 553) took 1695 ms. Trace:
I20260202 15:08:39.356841 14715 raft_consensus.cc:2749] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:39.379889 12643 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:39.381012 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:39.381466 12615 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:39.377952 14150 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.383025 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:39.384995 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.355875 12726 rpcz_store.cc:276] 0202 15:08:38.275848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.694978 (+419130us) service_pool.cc:225] Handling call
0202 15:08:38.695398 (+ 420us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:38.713202 (+ 17804us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.713242 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:38.713722 (+ 480us) log.cc:844] Serialized 26 byte log entry
0202 15:08:39.231058 (+517336us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:39.240553 (+ 9495us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:39.247441 (+ 6888us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:39.273805 (+ 26364us) raft_consensus.cc:1685] finished
0202 15:08:39.273826 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:39.317938 (+ 44112us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11318,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":13824,"thread_start_us":412330,"threads_started":1,"wal-append.queue_time_us":507181}
I20260202 15:08:39.389338 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:39.392359 12795 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:39.373889 14634 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:39.393599 14782 raft_consensus.cc:2749] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:39.394279 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.394469 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:39.397907 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.398025 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.398164 12793 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:39.399803 12796 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:39.391073 12791 raft_consensus.cc:1275] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260202 15:08:39.407186 14660 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:39.427198 14660 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:39.427603 14660 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:39.407305 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 63568.9us
I20260202 15:08:39.432255 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.432626 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.402232 14715 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:39.410846 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 554) took 1740 ms. Trace:
W20260202 15:08:39.410933 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 66248.1us
W20260202 15:08:39.412328 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 912158us
I20260202 15:08:39.412400 12614 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:39.415009 14619 consensus_queue.cc:1059] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:39.436444 14619 raft_consensus.cc:3055] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
W20260202 15:08:39.437292 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.436919 14619 raft_consensus.cc:740] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.437526 12690 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.438953 13361 raft_consensus.cc:1275] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:39.438375 14619 consensus_queue.cc:260] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.439273 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:39.439992 13354 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:39.440198 14619 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:39.442939 14715 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:39.441756 13355 rpcz_store.cc:276] 0202 15:08:37.650123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.734385 (+1084262us) service_pool.cc:225] Handling call
0202 15:08:38.764260 (+ 29875us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.787762 (+ 23502us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:39.207170 (+419408us) log.cc:844] Serialized 14 byte log entry
0202 15:08:39.319266 (+112096us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:39.325999 (+ 6733us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:39.353968 (+ 27969us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:39.372223 (+ 18255us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:39.390262 (+ 18039us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":118545,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":4833,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":133,"thread_start_us":383830,"threads_started":2}
I20260202 15:08:39.438959 12690 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.444870 12689 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.446403 14778 raft_consensus.cc:3055] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:39.447111 14778 raft_consensus.cc:740] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.416939 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.417312 12763 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:39.417914 14764 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:39.449298 14778 consensus_queue.cc:260] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.449980 14764 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:39.450464 14764 raft_consensus.cc:697] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.450634 14778 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.450994 14780 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:39.451484 14780 raft_consensus.cc:697] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.451354 14764 consensus_queue.cc:237] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.452315 14681 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:39.453724 12690 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.454488 14681 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.455451 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:39.457414 14258 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.458070 12643 raft_consensus.cc:1275] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260202 15:08:39.460153 14258 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:39.460359 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 98792.8us
I20260202 15:08:39.460494 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:39.461069 12646 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.462574 12643 raft_consensus.cc:1240] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:39.463169 14511 consensus_queue.cc:1059] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260202 15:08:39.464829 14688 raft_consensus.cc:3055] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:39.464973 14762 consensus_queue.cc:1048] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260202 15:08:39.388871 14176 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.419543 14787 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.340678 12720 rpcz_store.cc:276] 0202 15:08:34.826208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.168713 (+3342505us) service_pool.cc:225] Handling call
0202 15:08:38.199461 (+ 30748us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:38.225136 (+ 25675us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:38.235018 (+ 9882us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:38.319472 (+ 84454us) log.cc:844] Serialized 26 byte log entry
0202 15:08:38.975834 (+656362us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:39.161376 (+185542us) log.cc:844] Serialized 14 byte log entry
0202 15:08:39.217193 (+ 55817us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:39.240006 (+ 22813us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:39.244026 (+ 4020us) raft_consensus.cc:1685] finished
0202 15:08:39.244044 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:39.253589 (+ 9545us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36921,"prepare.queue_time_us":195771,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":56894,"raft.run_cpu_time_us":407,"raft.run_wall_time_us":25989,"thread_start_us":787357,"threads_started":2,"wal-append.queue_time_us":836623}
I20260202 15:08:39.467550 14176 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:39.468469 13356 raft_consensus.cc:1240] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:39.469103 14787 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.447326 13328 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:39.448097 14715 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:39.447824 14634 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260202 15:08:39.469744 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 519) took 2374 ms. Trace:
I20260202 15:08:39.469955 14715 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:39.378145 13359 rpcz_store.cc:276] 0202 15:08:37.649788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.702853 (+1053065us) service_pool.cc:225] Handling call
0202 15:08:38.732507 (+ 29654us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:38.747566 (+ 15059us) raft_consensus.cc:1534] Early marking committed up to index 7
0202 15:08:39.151254 (+403688us) log.cc:844] Serialized 14 byte log entry
0202 15:08:39.268470 (+117216us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:39.316376 (+ 47906us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:39.325020 (+ 8644us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:39.325430 (+ 410us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:39.345262 (+ 19832us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":158961,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":9364,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":176,"thread_start_us":432588,"threads_started":2,"wal-append.queue_time_us":246890}
I20260202 15:08:39.465219 14688 raft_consensus.cc:740] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.470109 13356 rpcz_store.cc:276] 0202 15:08:37.095475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:38.667655 (+1572180us) service_pool.cc:225] Handling call
0202 15:08:38.667923 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:39.468890 (+800967us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:39.469471 (+ 581us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b54005ec4d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:39.469672 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1840283520}
I20260202 15:08:39.416596 13358 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:39.462324 12765 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.422446 13362 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:39.471613 12690 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.458689 12544 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45928 (request call id 317) took 6594 ms. Trace:
I20260202 15:08:39.432986 14649 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:39.470413 14715 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:39.442201 12615 leader_election.cc:304] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:39.493695 12646 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
W20260202 15:08:39.494423 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 82212.2us
I20260202 15:08:39.474877 14176 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:39.497148 12688 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:39.484751 14762 raft_consensus.cc:2749] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:39.503100 13328 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:39.500838 14258 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:39.460145 12545 catalog_manager.cc:5654] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 5, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 5 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:39.504304 14258 raft_consensus.cc:697] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:39.506515 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 64504.6us
I20260202 15:08:39.508271 14306 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:39.510182 14306 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:39.511557 12764 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:39.512737 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.508152 14258 consensus_queue.cc:237] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.512487 14693 consensus_queue.cc:1059] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260202 15:08:39.502610 14634 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:39.514194 14793 raft_consensus.cc:3055] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:39.496130 14511 consensus_queue.cc:1059] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260202 15:08:39.513665 14693 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:39.507014 12689 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:39.518940 14691 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:39.519359 14691 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:39.519691 14691 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:39.483839 12544 rpcz_store.cc:276] 0202 15:08:32.864680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.871133 (+ 6453us) service_pool.cc:225] Handling call
0202 15:08:39.458485 (+6587352us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:08:38.915046 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:08:38.915780 (+ 734us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:08:38.915829 (+ 49us) write_op.cc:435] Acquired schema lock
0202 15:08:38.915854 (+ 25us) tablet.cc:660] Decoding operations
0202 15:08:38.916731 (+ 877us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:08:38.916804 (+ 73us) write_op.cc:641] Partition lock acquired for write op
0202 15:08:38.916838 (+ 34us) tablet.cc:683] Acquiring locks for 5 operations
0202 15:08:38.917446 (+ 608us) tablet.cc:699] Row locks acquired
0202 15:08:38.917470 (+ 24us) write_op.cc:260] PREPARE: finished
0202 15:08:38.917623 (+ 153us) write_op.cc:270] Start()
0202 15:08:38.917749 (+ 126us) write_op.cc:276] Timestamp: P: 1770044918917572 usec, L: 0
0202 15:08:38.917769 (+ 20us) op_driver.cc:348] REPLICATION: starting
0202 15:08:38.924782 (+ 7013us) log.cc:844] Serialized 2147 byte log entry
0202 15:08:38.927357 (+ 2575us) op_driver.cc:464] REPLICATION: finished
0202 15:08:39.320235 (+392878us) write_op.cc:301] APPLY: starting
0202 15:08:39.320341 (+ 106us) tablet.cc:1365] starting BulkCheckPresence
0202 15:08:39.320506 (+ 165us) tablet.cc:1368] finished BulkCheckPresence
0202 15:08:39.320624 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:08:39.349904 (+ 29280us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:08:39.350239 (+ 335us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0202 15:08:39.350319 (+ 80us) write_op.cc:312] APPLY: finished
0202 15:08:39.350721 (+ 402us) log.cc:844] Serialized 47 byte log entry
0202 15:08:39.350973 (+ 252us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:39.351220 (+ 247us) write_op.cc:454] Released schema lock
0202 15:08:39.351290 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0202 15:08:39.351876 (+ 586us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:39.351914 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2612355,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":392572,"apply.run_cpu_time_us":3695,"apply.run_wall_time_us":31643,"num_ops":5,"prepare.queue_time_us":282297,"prepare.run_cpu_time_us":3946,"prepare.run_wall_time_us":10084,"replication_time_us":9423,"thread_start_us":559163,"threads_started":2}]]}
W20260202 15:08:39.521967 14502 consensus_meta.cc:287] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.624s user 0.009s sys 0.007s
I20260202 15:08:39.522377 14502 raft_consensus.cc:2749] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:39.522993 12545 catalog_manager.cc:5654] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:39.525065 14511 consensus_queue.cc:1243] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260202 15:08:39.525588 14468 raft_consensus.cc:3055] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:39.525947 14468 raft_consensus.cc:740] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.526371 14535 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:39.529691 14535 raft_consensus.cc:697] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:39.526822 14468 consensus_queue.cc:260] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.531558 14634 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:39.532043 14634 raft_consensus.cc:697] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:39.533007 14634 consensus_queue.cc:237] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.530784 14535 consensus_queue.cc:237] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.532205 14681 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:39.530334 14150 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.530746 14468 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:39.517103 14793 raft_consensus.cc:740] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.536069 14150 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.535707 14793 consensus_queue.cc:260] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.537200 14793 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.537986 12545 catalog_manager.cc:5654] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 3 to 4, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 4 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260202 15:08:39.541612 14797 log.cc:927] Time spent T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.130s user 0.001s sys 0.000s
I20260202 15:08:39.473335 14688 consensus_queue.cc:260] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.547744 14722 consensus_queue.cc:1059] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:39.545224 14564 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:39.549152 14782 raft_consensus.cc:3055] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:39.506229 14176 raft_consensus.cc:697] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:39.549554 14782 raft_consensus.cc:740] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.550606 14782 consensus_queue.cc:260] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.552067 14688 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.550427 14176 consensus_queue.cc:237] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.546146 12615 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:39.546864 14724 consensus_queue.cc:1059] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:39.554587 14782 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.452482 14780 consensus_queue.cc:237] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:39.578474 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 116274us
I20260202 15:08:39.555958 14722 raft_consensus.cc:3055] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:39.583271 14722 raft_consensus.cc:740] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.584268 13362 raft_consensus.cc:1275] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:39.584160 14722 consensus_queue.cc:260] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:39.587178 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 127099us
I20260202 15:08:39.588003 14722 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.588513 13357 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:39.589597 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.580926 14724 raft_consensus.cc:2749] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:39.591095 13356 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:39.593266 13358 raft_consensus.cc:1240] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 1. Current term is 2. Ops: []
I20260202 15:08:39.599123 14778 consensus_queue.cc:1048] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:39.600409 14778 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:08:39.601586 14778 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260202 15:08:39.602842 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.600425 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:39.604322 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:39.647303 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.674650 14770 consensus_queue.cc:1059] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.061s
I20260202 15:08:39.675150 12643 raft_consensus.cc:2410] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:39.675870 14770 raft_consensus.cc:3055] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:39.676280 14770 raft_consensus.cc:740] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:39.677261 14770 consensus_queue.cc:260] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:39.678608 14805 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
W20260202 15:08:39.679682 14673 consensus_queue.cc:1175] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "bbf140958c3b4f339c512a9dc4be2dab" responder_term: 5 status { last_received { term: 5 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 5 index: 6 } }
I20260202 15:08:39.680431 12545 catalog_manager.cc:5654] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 3 to 4, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 4 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:39.682272 14770 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:39.649140 14780 consensus_queue.cc:1048] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260202 15:08:39.696278 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.696465 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.697113 12793 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:39.697255 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.697589 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.698355 12726 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:39.698256 13328 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:39.683221 14619 consensus_queue.cc:1059] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20260202 15:08:39.728636 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.699635 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.710224 12690 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:39.702972 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.699708 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.709920 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:39.743072 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 101664us
I20260202 15:08:39.727744 12545 catalog_manager.cc:5654] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 4, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 4 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:39.699364 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:39.748066 12795 raft_consensus.cc:2368] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:39.697278 12791 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:39.713328 12793 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:39.704195 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.756474 12545 catalog_manager.cc:5654] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 3, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:39.764312 14724 consensus_queue.cc:1059] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:39.765134 14724 consensus_queue.cc:1059] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:39.765673 14782 raft_consensus.cc:3055] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:39.765374 14691 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.765946 14724 raft_consensus.cc:3055] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:39.704936 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.766371 14724 raft_consensus.cc:740] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.766114 14782 raft_consensus.cc:740] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.766829 12717 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:39.772347 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 74158.8us
I20260202 15:08:39.770061 12545 catalog_manager.cc:5654] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:39.768155 12764 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.767076 12719 raft_consensus.cc:2393] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:39.773491 12615 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:39.777143 14258 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:39.710379 14762 consensus_queue.cc:1059] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260202 15:08:39.692437 14468 consensus_queue.cc:1059] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260202 15:08:39.778973 14681 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:39.711357 14400 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.804708 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.804819 12796 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:39.779795 14722 consensus_queue.cc:1059] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:39.785672 14809 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:39.786679 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.788220 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.787017 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.789006 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.789822 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.790943 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.688728 14601 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260202 15:08:39.767619 14306 raft_consensus.cc:1064] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:39.798152 12545 catalog_manager.cc:5654] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 3, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:39.810014 13328 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:39.819242 12545 catalog_manager.cc:5654] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:39.820178 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.822086 12724 raft_consensus.cc:1240] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:39.823014 12725 raft_consensus.cc:3060] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:39.826972 14176 raft_consensus.cc:2764] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260202 15:08:39.827205 12721 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:39.828668 12721 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260202 15:08:39.833335 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:39.833701 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:39.833935 12615 connection.cc:232] RPC call timeout handler was delayed by -1.0065775209998264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2194s for a response
I20260202 15:08:39.834900 14724 consensus_queue.cc:260] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.835482 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:39.836004 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 152412us
I20260202 15:08:39.836412 12722 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:39.836800 14602 raft_consensus.cc:3055] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:39.837204 14602 raft_consensus.cc:740] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:39.837488 14502 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:39.837929 14502 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.837963 14602 consensus_queue.cc:260] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.841172 14793 raft_consensus.cc:3055] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:39.819257 14762 raft_consensus.cc:3055] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:39.841693 14793 raft_consensus.cc:740] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.841827 14762 raft_consensus.cc:740] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.842670 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:39.842548 14793 consensus_queue.cc:260] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.843390 12643 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:39.844797 12763 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:39.828469 14778 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260202 15:08:39.851647 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 83620.7us
I20260202 15:08:39.853435 12764 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.862880 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.868363 12726 raft_consensus.cc:2393] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:39.869141 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.869378 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.870491 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.871243 12719 raft_consensus.cc:1240] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:39.871637 12721 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.872731 12719 raft_consensus.cc:1275] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260202 15:08:39.858120 14619 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:39.874348 14619 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:39.874730 14619 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:39.879667 14794 raft_consensus.cc:3055] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:39.880059 14794 raft_consensus.cc:740] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.881069 14511 raft_consensus.cc:2749] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:39.880949 14794 consensus_queue.cc:260] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.882356 14794 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.881989 14688 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260202 15:08:39.881932 14782 consensus_queue.cc:260] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.887878 14782 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.844552 14150 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:39.894754 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:39.895920 14660 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.896543 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.897317 12724 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 3: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:39.897527 12645 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:39.897929 14660 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:39.897864 14619 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.900327 14619 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:39.890872 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:39.858453 14681 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:39.900784 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:39.901335 12643 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:39.901057 14681 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.901706 14724 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.894135 14502 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:39.904510 14681 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:39.893496 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:39.906023 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:39.902287 12725 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:39.906019 12719 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.907708 12720 raft_consensus.cc:1240] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:39.910596 14691 leader_election.cc:290] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.911144 14400 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:39.912189 12647 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
W20260202 15:08:39.889559 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 79598.6us
I20260202 15:08:39.840389 14258 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:39.900305 14306 raft_consensus.cc:1068] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Unable to promote non-voter 66b8c715b9e6479e9218b08bd3156380: Illegal state: Leader has not yet committed an operation in its own term
I20260202 15:08:39.918386 12723 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:39.918524 12764 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.920794 12722 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:39.847654 14649 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:39.922006 12718 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:39.925081 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.863426 14602 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:39.928889 13328 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.932528 12645 raft_consensus.cc:1240] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:39.932886 12615 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:39.935061 13362 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:39.912688 14400 consensus_queue.cc:1059] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260202 15:08:39.940137 14306 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:39.941618 14306 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:39.941985 14306 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:39.935155 14502 raft_consensus.cc:2804] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:08:39.947278 14400 raft_consensus.cc:3055] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:39.947393 12723 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:39.948606 12722 raft_consensus.cc:1240] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260202 15:08:39.935595 14809 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:39.924742 14619 consensus_queue.cc:1059] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260202 15:08:39.954248 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:39.947880 14258 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.936324 14535 consensus_queue.cc:1059] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260202 15:08:39.950536 14502 raft_consensus.cc:697] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.947680 14400 raft_consensus.cc:740] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.938161 14649 raft_consensus.cc:697] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
W20260202 15:08:39.957969 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.940572 14793 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 1 FOLLOWER]: Advancing to term 2
W20260202 15:08:39.961274 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:39.962895 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:39.961541 12764 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.958669 14176 raft_consensus.cc:3055] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:39.956070 14619 raft_consensus.cc:3055] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:39.964135 14176 raft_consensus.cc:740] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:39.964311 14619 raft_consensus.cc:740] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:39.965023 14176 consensus_queue.cc:260] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.966148 14176 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:39.967267 14502 consensus_queue.cc:237] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.969074 14771 consensus_queue.cc:1059] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:39.969336 12690 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.970045 14771 raft_consensus.cc:3055] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:39.970405 14771 raft_consensus.cc:740] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:39.960876 14258 consensus_queue.cc:1048] T 85ea3f0239bb4603a3a363108154ef77 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.010s
I20260202 15:08:39.961010 14715 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.973668 14715 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:39.974711 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:39.975112 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:39.975654 13358 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:39.979218 14306 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.930038 14762 consensus_queue.cc:260] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.983623 14762 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:39.985260 14649 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.985965 12721 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:39.989053 12647 raft_consensus.cc:1275] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260202 15:08:39.982189 14306 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:39.978421 12719 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:39.991039 13328 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:39.979674 14400 consensus_queue.cc:260] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:39.992934 12645 raft_consensus.cc:1240] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:39.991829 14770 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:39.994868 14770 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:39.997505 14770 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:39.998088 14809 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:39.998562 14809 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:39.998981 14809 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:40.004871 13362 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:40.005772 12722 raft_consensus.cc:1240] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
W20260202 15:08:40.006394 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 44910.8us
I20260202 15:08:40.007959 12765 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.008374 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:40.008680 14150 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.009107 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:40.009259 12615 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:40.014114 12721 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:40.014631 14306 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.015491 12615 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.017218 12615 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.018000 14649 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:40.018671 14150 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:40.019693 13328 leader_election.cc:400] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:40.019971 13328 leader_election.cc:403] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260202 15:08:40.019693 12765 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.021459 13328 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.021657 14150 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:40.022627 12615 leader_election.cc:400] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3.
I20260202 15:08:40.023020 12615 leader_election.cc:403] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260202 15:08:40.023208 14150 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:40.023523 14802 raft_consensus.cc:2749] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3.
I20260202 15:08:40.024178 14778 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:39.971166 14771 consensus_queue.cc:260] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.025442 12765 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.002966 14400 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:40.027403 14649 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.027714 12764 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.027868 12643 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:40.028657 14306 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:40.029440 14619 consensus_queue.cc:260] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:40.030861 14770 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:40.030941 12763 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:40.031615 14770 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:40.032847 13328 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.034179 12721 raft_consensus.cc:2410] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260202 15:08:40.035585 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:40.036417 12722 raft_consensus.cc:2410] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260202 15:08:40.036762 14681 consensus_queue.cc:1059] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:40.038080 13358 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:40.039069 12689 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.039655 12717 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:40.039983 14634 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:40.040509 14634 raft_consensus.cc:697] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:40.041474 14634 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.043061 14535 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:40.033591 14770 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:40.043380 14693 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.043752 14770 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:40.043099 12764 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.044181 14693 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:40.044744 14688 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:40.044862 14693 raft_consensus.cc:697] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:40.045229 14688 raft_consensus.cc:697] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:40.045867 14693 consensus_queue.cc:237] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.046162 14688 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:40.043273 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 74047.2us
I20260202 15:08:40.006232 14258 consensus_queue.cc:1048] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
I20260202 15:08:40.039887 14771 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:40.043658 14535 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.055585 12690 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.056584 14715 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:40.057229 14535 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:40.058120 12690 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.059087 14715 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:40.060748 12690 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:40.029443 14814 log.cc:927] Time spent T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260202 15:08:40.063665 12765 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.062399 14511 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.067634 14715 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:40.068186 14619 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:40.070828 14778 consensus_queue.cc:1059] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:40.071729 14306 raft_consensus.cc:3055] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:40.072077 14306 raft_consensus.cc:740] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:40.034278 14604 raft_consensus.cc:2749] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:40.079861 12690 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.080497 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:40.075615 14762 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:40.079871 12796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:40.081810 12796 raft_consensus.cc:606] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.082201 12796 raft_consensus.cc:614] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260202 15:08:40.072870 14306 consensus_queue.cc:260] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.088155 14306 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:40.088791 14511 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.089300 14511 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.081033 13356 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 5
I20260202 15:08:40.074522 14535 raft_consensus.cc:2749] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:40.073509 14724 consensus_queue.cc:1059] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260202 15:08:40.108843 12793 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:40.109503 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:40.110280 12795 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:40.115216 12690 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.115769 14794 consensus_queue.cc:1059] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:40.098235 14771 raft_consensus.cc:2749] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.073837 14809 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.125276 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:40.126250 12796 raft_consensus.cc:2410] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260202 15:08:40.126133 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 103469us
I20260202 15:08:40.105111 14764 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:40.129182 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:40.131688 12765 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.134739 13328 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.125851 14715 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:40.135778 12795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:40.154196 12795 raft_consensus.cc:606] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:40.157788 12615 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.158638 14511 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:40.160352 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:40.160583 12615 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.162369 14724 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:40.163470 12764 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.170043 14724 raft_consensus.cc:2749] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.137851 14400 raft_consensus.cc:3055] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:40.174230 14400 raft_consensus.cc:740] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:40.171494 14306 raft_consensus.cc:3055] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:40.174856 14306 raft_consensus.cc:740] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:40.175206 14400 consensus_queue.cc:260] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.175582 14306 consensus_queue.cc:260] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.176637 14400 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:40.176769 14306 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:40.177870 12795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1"
dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:35714
I20260202 15:08:40.178380 12795 raft_consensus.cc:606] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:40.140039 14809 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:40.146847 14764 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.147174 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:40.148456 12765 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.149017 14688 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:40.150309 13358 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:40.133509 14762 consensus_queue.cc:1048] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260202 15:08:40.152348 14770 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.208320 14502 raft_consensus.cc:993] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:40.209605 14770 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:40.209785 14502 raft_consensus.cc:1081] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:40.209836 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:40.210125 13356 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:40.210454 12645 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:40.170542 14715 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:40.211544 14715 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.211861 13328 leader_election.cc:400] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:40.212162 13328 leader_election.cc:403] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260202 15:08:40.212787 13328 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.213187 14715 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:40.213438 14809 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:40.213864 13328 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.214115 14809 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260202 15:08:40.214910 14809 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:40.214951 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:40.161664 12615 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:40.172415 14511 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:40.215235 12764 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:40.216096 14511 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:40.216440 14793 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:40.216579 14511 raft_consensus.cc:697] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:40.217489 14649 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:40.217583 14511 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.174271 14535 raft_consensus.cc:2749] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.138629 13357 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:08:40.187258 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260202 15:08:40.219081 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:40.219661 12646 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:40.220458 12688 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:40.215483 14809 raft_consensus.cc:697] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:40.198930 14764 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:40.199904 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 51543.5us
I20260202 15:08:40.227329 12765 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.200539 14688 raft_consensus.cc:2804] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:40.173910 14778 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.234360 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:40.136111 12690 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:40.224829 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 63236us
I20260202 15:08:40.234697 14764 raft_consensus.cc:2804] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:40.235430 13357 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44934
I20260202 15:08:40.235512 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:40.235847 14762 consensus_queue.cc:1059] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:08:40.236603 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:40.236850 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:40.237183 12793 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:40.237318 14762 raft_consensus.cc:3055] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:40.237695 14762 raft_consensus.cc:740] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:40.237888 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:40.231470 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:40.127674 14604 consensus_queue.cc:1059] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260202 15:08:40.236064 13357 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260202 15:08:40.231902 12721 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:40.238504 12717 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:40.256274 13328 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.235523 12765 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.256500 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:40.257907 12725 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:40.256814 12645 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:40.245445 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:40.258745 12798 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:40.240190 14602 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260202 15:08:40.234786 14688 raft_consensus.cc:697] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:40.261487 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:40.264132 12718 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:40.264400 12615 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.298723 12794 raft_consensus.cc:2393] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:40.275707 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:40.300964 12796 raft_consensus.cc:2410] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260202 15:08:40.239187 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:40.240474 14634 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:40.298158 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:40.239775 14764 raft_consensus.cc:697] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:40.260411 13357 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:40.298193 14176 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:40.260828 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:40.232189 14782 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260202 15:08:40.261099 12720 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
W20260202 15:08:40.309398 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 45087.5us
I20260202 15:08:40.316838 12615 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.239951 14802 consensus_queue.cc:1059] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260202 15:08:40.315227 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:40.319775 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 190819us
I20260202 15:08:40.302435 14693 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:40.303336 14636 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.238564 14762 consensus_queue.cc:260] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.304929 14634 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:40.303535 14794 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:40.222635 14809 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:40.325026 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 95620.2us
I20260202 15:08:40.261667 14691 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260202 15:08:40.327250 14794 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.328037 14783 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:40.325801 14634 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:40.332921 13356 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:40.333565 13362 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:40.343040 12615 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.336351 12795 raft_consensus.cc:1240] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:40.345119 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:40.345685 12726 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:40.348740 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:40.336705 13328 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.321151 12792 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:40.319391 14778 raft_consensus.cc:2749] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.341594 12725 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:40.315783 12791 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:40.354686 12724 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:40.368808 14722 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260202 15:08:40.369997 14771 raft_consensus.cc:3055] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260202 15:08:40.370429 14771 raft_consensus.cc:740] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:40.371563 14724 raft_consensus.cc:3055] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:40.371982 14724 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:40.371477 14771 consensus_queue.cc:260] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.372668 14724 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.373095 14771 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:40.373728 14724 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:40.375777 12793 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:40.376202 14764 consensus_queue.cc:237] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.379179 14468 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.379640 14468 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.380117 12797 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:40.381351 14468 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:40.322646 14688 consensus_queue.cc:237] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.386790 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:40.387185 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 50580.7us
I20260202 15:08:40.398514 13328 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:40.411007 12763 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:40.417958 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:40.370332 14722 consensus_queue.cc:1059] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260202 15:08:40.385927 14783 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260202 15:08:40.425812 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 99437.3us
I20260202 15:08:40.427958 13328 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.428604 14176 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:40.398515 14770 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.388342 14740 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:40.402388 14634 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:40.439620 14827 log.cc:927] Time spent T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260202 15:08:40.456903 12765 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:40.459584 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 93655.7us
I20260202 15:08:40.461311 12615 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:40.434152 14564 raft_consensus.cc:3055] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:40.464313 12690 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:40.466059 12723 raft_consensus.cc:1240] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:40.466964 13328 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.463781 14740 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.464900 14564 raft_consensus.cc:740] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:40.583137 12797 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:40.471191 14787 consensus_queue.cc:1059] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260202 15:08:40.577971 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:40.587253 12798 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:40.578598 12726 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
W20260202 15:08:40.579027 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 114782us
I20260202 15:08:40.455847 14673 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:40.461956 14770 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260202 15:08:40.465975 14783 consensus_queue.cc:1059] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260202 15:08:40.466593 14722 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election won for term 4
W20260202 15:08:40.588129 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 125274us
I20260202 15:08:40.595315 12645 raft_consensus.cc:2410] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:40.597626 12615 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:40.481727 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:40.604115 12720 raft_consensus.cc:2410] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:40.605309 12792 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:40.606125 14306 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.506989 14764 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.577702 14811 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.627024 14649 raft_consensus.cc:3055] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:40.627480 14649 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:40.637468 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:40.641702 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 184848us
I20260202 15:08:40.643327 13360 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:40.616214 14809 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:40.591688 14660 consensus_queue.cc:1059] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260202 15:08:40.618032 14794 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:40.651156 12718 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:40.652504 14783 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:40.606638 14306 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.657464 12690 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:40.676133 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:40.653916 14783 raft_consensus.cc:697] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
W20260202 15:08:40.686300 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 88732.4us
I20260202 15:08:40.687361 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:40.688324 12646 raft_consensus.cc:2393] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:40.688277 12615 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:40.659355 12793 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:40.686947 14844 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260202 15:08:40.696868 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 50223.8us
I20260202 15:08:40.655434 14660 consensus_queue.cc:1243] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 6921687ed3814cccb2bd9c606875b9f0 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:08:40.656219 14782 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.699719 13360 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260202 15:08:40.715111 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 89830.7us
I20260202 15:08:40.655907 12717 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:40.656623 14176 raft_consensus.cc:697] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:40.699502 12615 leader_election.cc:304] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:40.759562 12690 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:40.690662 14839 log.cc:927] Time spent T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20260202 15:08:40.760154 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 60728.8us
I20260202 15:08:40.692113 13356 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:40.769099 12763 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:40.772451 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 60460.5us
I20260202 15:08:40.698619 14722 raft_consensus.cc:697] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:40.774228 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:40.776667 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 295604us
I20260202 15:08:40.684119 14793 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260202 15:08:40.777560 13328 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.639580 14564 consensus_queue.cc:260] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.632568 14811 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:40.767516 14838 log.cc:927] Time spent T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260202 15:08:40.784613 12724 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:08:40.771920 14511 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:40.771447 14634 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:40.765092 14835 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.771522 14636 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260202 15:08:40.780563 14828 log.cc:927] Time spent T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.168s user 0.005s sys 0.010s
I20260202 15:08:40.799361 13362 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:40.785487 12797 raft_consensus.cc:1240] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:40.783059 14832 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.765120 14782 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.788559 14673 raft_consensus.cc:697] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:40.776160 14258 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
I20260202 15:08:40.814429 12690 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:40.691195 14783 consensus_queue.cc:237] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.830183 12646 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260202 15:08:40.831156 14850 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:40.831295 14845 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:40.831818 14850 raft_consensus.cc:697] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:40.832000 14845 raft_consensus.cc:697] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:40.815408 12793 raft_consensus.cc:1240] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:40.816128 14674 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.816025 14770 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:40.813046 14740 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:40.842769 12690 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:40.844261 14673 consensus_queue.cc:237] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.840224 14674 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.840164 14835 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:40.847550 14835 raft_consensus.cc:697] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:40.847841 14674 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:40.848183 14829 log.cc:927] Time spent T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.360s user 0.001s sys 0.000s
I20260202 15:08:40.848325 14835 consensus_queue.cc:237] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.850354 14306 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:40.844671 14840 log.cc:927] Time spent T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
I20260202 15:08:40.859827 12647 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:40.815598 14794 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260202 15:08:40.861868 12688 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:40.832890 14850 consensus_queue.cc:237] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.868990 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:40.869269 14681 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:40.869729 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:40.855746 14564 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:40.878139 14847 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260202 15:08:40.860536 14809 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:40.833050 14845 consensus_queue.cc:237] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:40.867612 14634 consensus_queue.cc:1048] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260202 15:08:40.855943 13354 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:40.819079 14176 consensus_queue.cc:237] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.916447 12798 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:40.916648 12718 raft_consensus.cc:1275] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:40.923260 12794 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
W20260202 15:08:40.924222 12763 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:40.927891 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:40.934391 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 144886us
I20260202 15:08:40.947707 13328 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:40.934638 14681 raft_consensus.cc:697] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:40.939088 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:40.954165 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:40.940069 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:40.944840 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 109663us
I20260202 15:08:40.861313 14770 raft_consensus.cc:697] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:40.951895 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:40.936112 13357 consensus_meta.cc:287] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.615s user 0.006s sys 0.006s
I20260202 15:08:40.969807 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:40.970578 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:40.839854 14722 consensus_queue.cc:237] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:40.974167 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:40.978856 12794 raft_consensus.cc:2393] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:40.979110 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:40.981577 12798 raft_consensus.cc:2393] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 2: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:40.977360 13363 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:40.984414 12725 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:40.955506 14306 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.952960 14793 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:40.970217 14660 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.970079 14740 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:40.985153 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:41.009863 12794 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
W20260202 15:08:40.976240 14724 consensus_meta.cc:287] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.602s user 0.006s sys 0.003s
W20260202 15:08:41.001533 12721 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.020s sys 0.007s
I20260202 15:08:40.977120 12793 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:41.016706 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:41.017692 12721 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s user 0.021s sys 0.008s
I20260202 15:08:40.980582 14258 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.981834 14782 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.033696 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:40.980793 14832 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:40.985870 14794 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.034299 12794 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:41.007895 14811 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:41.039620 12798 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260202 15:08:41.040019 14852 log.cc:927] Time spent T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.178s user 0.001s sys 0.000s
I20260202 15:08:40.976912 13357 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:40.975569 14849 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260202 15:08:41.021034 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.015686 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:41.019302 14854 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:41.030294 14740 raft_consensus.cc:697] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:08:41.050882 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 68436us
I20260202 15:08:41.015394 14306 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.016134 14660 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.034988 14793 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.046873 14604 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.103432 14306 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.034277 14258 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.077932 14636 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:41.125382 12644 raft_consensus.cc:3055] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:41.044703 14794 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.127434 12725 raft_consensus.cc:1275] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:41.104358 12614 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:41.078508 13356 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
I20260202 15:08:41.085671 12643 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:41.134270 12644 raft_consensus.cc:740] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:41.094574 14673 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:41.143443 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 159776us
I20260202 15:08:41.124878 13355 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
W20260202 15:08:41.152921 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 59853.6us
I20260202 15:08:41.157155 12690 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:41.158854 14306 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:41.138072 12720 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:41.051553 14770 consensus_queue.cc:237] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.163605 12690 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:41.159435 14306 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.060660 14681 consensus_queue.cc:237] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:41.169466 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 65181.4us
I20260202 15:08:41.125923 14511 consensus_queue.cc:1048] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260202 15:08:41.172708 12614 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:41.173028 14306 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:41.175469 14306 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:41.175849 14306 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.176138 14306 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:41.175933 14845 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:08:41.180365 14306 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.179653 14845 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:08:41.181546 14306 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.151628 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.182744 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:41.167516 12644 consensus_queue.cc:260] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.094528 14854 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.078141 14832 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.183516 12798 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:41.196084 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.198737 12722 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:41.193039 14850 raft_consensus.cc:1064] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:41.137531 14604 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:41.203325 14850 raft_consensus.cc:1068] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Unable to promote non-voter 66b8c715b9e6479e9218b08bd3156380: Illegal state: Leader has not yet committed an operation in its own term
I20260202 15:08:41.203657 14856 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.203882 14873 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:41.204216 14856 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.205760 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.206187 14856 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.206506 12725 raft_consensus.cc:2393] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 2: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:41.206674 14862 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.207154 14862 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.207742 12795 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:41.208070 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:41.208550 12722 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:41.208845 14862 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:41.211335 14859 log.cc:927] Time spent T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260202 15:08:41.204694 14873 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:41.154992 14673 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.221501 12722 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:41.174016 14782 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.197292 14468 consensus_queue.cc:1048] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20260202 15:08:41.198078 14845 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260202 15:08:41.246892 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 74240.1us
I20260202 15:08:41.247551 12615 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:41.248955 14845 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.227231 14782 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.252403 13357 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:41.297065 12648 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:41.297370 14878 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:41.297416 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:41.298133 12650 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:41.297984 14878 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.266520 14856 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.308872 12688 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:41.266584 14846 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:41.267259 14511 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.278363 14850 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:41.281787 14845 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:41.317729 13326 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:41.291185 14847 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:41.184985 14740 consensus_queue.cc:237] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:41.344000 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 140578us
I20260202 15:08:41.301471 14793 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:41.343254 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 170542us
I20260202 15:08:41.281011 14862 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
W20260202 15:08:41.348896 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:41.319664 12720 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260202 15:08:41.316301 14660 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.318560 14258 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.325443 14850 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.340037 14869 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.340333 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:44934 (request call id 548) took 1075 ms. Trace:
I20260202 15:08:41.421892 14850 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:41.432296 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:41.432744 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1695) took 1182 ms. Trace:
I20260202 15:08:41.428718 14867 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.429222 14864 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260202 15:08:41.428824 14649 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.348537 14849 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:41.354250 14634 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260202 15:08:41.431597 12763 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:41.344190 14794 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:41.315905 14856 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:41.436847 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:41.433079 12723 rpcz_store.cc:276] 0202 15:08:40.250775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.250905 (+ 130us) service_pool.cc:225] Handling call
0202 15:08:40.262031 (+ 11126us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:40.466558 (+204527us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:41.432554 (+965996us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54002fe754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:41.432691 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":468802432}
I20260202 15:08:41.338881 14724 consensus_queue.cc:1059] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.084s
I20260202 15:08:41.357442 14793 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.455281 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.455955 12726 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:41.438850 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.457127 12765 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:41.440521 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:41.440743 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:41.441208 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:41.324542 14511 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.447163 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:41.445237 14258 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.449219 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.449301 14604 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.452493 14870 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
W20260202 15:08:41.453452 14877 log.cc:927] Time spent T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
W20260202 15:08:41.454196 14874 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260202 15:08:41.523094 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:41.454944 14793 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.527283 12649 raft_consensus.cc:2410] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:41.438004 14783 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.457975 12648 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:41.445680 13354 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:41.346196 14847 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.514506 14770 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.447199 14849 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.448792 14634 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.430294 13357 rpcz_store.cc:276] 0202 15:08:40.232813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.235311 (+ 2498us) service_pool.cc:225] Handling call
0202 15:08:41.301338 (+1066027us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:41.516455 12642 raft_consensus.cc:2410] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:41.519047 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:41.457998 14649 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:41.521301 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 74602.4us
W20260202 15:08:41.521930 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 358378us
I20260202 15:08:41.495421 14780 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:41.445327 14867 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:41.525683 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 68601.8us
I20260202 15:08:41.513948 13363 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:41.454972 14880 log.cc:927] Time spent T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20260202 15:08:41.554597 12763 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:41.553646 12641 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:41.525578 14673 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:41.558290 12690 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:41.526008 14832 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.532317 14862 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:41.525728 14854 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:41.443383 14869 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.531029 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.519644 14258 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.554771 14849 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:41.555467 14883 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.556080 14762 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:41.572460 14862 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.555127 14770 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.558605 14782 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.563189 14780 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.575855 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.526618 14870 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.588152 14832 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:41.589636 14865 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.590173 14889 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.644663 14889 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:41.646212 14886 log.cc:927] Time spent T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20260202 15:08:41.604687 14862 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:41.573421 14878 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.599004 12720 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:41.599792 14511 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:41.656958 12763 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:41.601982 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:41.552634 14783 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.632652 13363 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:41.634733 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.666846 12722 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:41.667691 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.668174 12722 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:41.637974 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.579736 14864 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:41.590649 14861 log.cc:927] Time spent T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.263s user 0.000s sys 0.001s
I20260202 15:08:41.646605 14889 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.636744 14888 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260202 15:08:41.680418 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:41.680967 12649 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
I20260202 15:08:41.681197 12615 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:41.641769 14872 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.642449 14258 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:41.642927 14894 log.cc:927] Time spent T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260202 15:08:41.643196 13357 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:08:41.643895 14891 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260202 15:08:41.643797 14865 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:41.590844 14863 log.cc:927] Time spent T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.333s user 0.004s sys 0.007s
W20260202 15:08:41.616288 14564 consensus_meta.cc:287] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.684s user 0.005s sys 0.005s
I20260202 15:08:41.604357 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.629046 14902 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.553730 14634 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:41.669477 12765 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:41.671576 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.700392 13363 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:41.629422 14794 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.701665 12689 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:41.672312 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.679706 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.703579 12722 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:41.684156 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:41.601282 14883 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.671897 14885 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:41.704162 12641 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:41.667045 14856 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.732085 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.732918 12614 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:41.787945 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:41.788200 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 107061us
I20260202 15:08:41.788517 12798 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 2.
W20260202 15:08:41.788745 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 55950.4us
I20260202 15:08:41.687500 14782 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.731040 14794 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.785873 12718 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260202 15:08:41.709659 14258 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:41.749187 12645 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:08:41.682636 13354 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:41.787334 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.726217 13361 raft_consensus.cc:2410] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:41.799746 12722 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:41.787976 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:41.790594 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:41.791076 13363 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:41.793155 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1415) took 1518 ms. Trace:
I20260202 15:08:41.801635 14913 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.802238 14913 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.803457 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:41.731231 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 549) took 1464 ms. Trace:
I20260202 15:08:41.746203 14888 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:41.706415 14885 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.802759 14794 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:41.804073 12800 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:41.699841 14864 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.748811 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.785431 14883 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.715430 14902 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.747593 14865 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.804936 12795 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:41.833936 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.833410 12724 raft_consensus.cc:1275] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260202 15:08:41.812206 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.789508 14869 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.787179 14847 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:41.736084 14793 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:41.819200 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:41.959627 12798 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:41.868194 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1496) took 1179 ms. Trace:
I20260202 15:08:41.838373 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.837472 12615 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:41.950783 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.950158 12722 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
W20260202 15:08:41.859934 12793 consensus_meta.cc:287] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.823s user 0.010s sys 0.000s
I20260202 15:08:41.745010 14867 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:41.850384 14903 log.cc:927] Time spent T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260202 15:08:41.957008 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:41.799733 14258 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.697206 14872 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.961021 12646 rpcz_store.cc:276] 0202 15:08:40.689111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.689242 (+ 131us) service_pool.cc:225] Handling call
0202 15:08:40.689510 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:40.830593 (+141083us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:41.867959 (+1037366us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b540020dd54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:41.868123 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316617984}
I20260202 15:08:41.961794 12765 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:41.835253 14176 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:41.961570 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.748051 14908 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260202 15:08:41.962904 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 125511us
I20260202 15:08:41.804312 14673 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:41.860349 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.812970 14511 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.990141 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.991007 12724 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:41.954842 14770 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:41.805289 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:41.850428 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1787) took 1087 ms. Trace:
I20260202 15:08:41.981201 12722 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:41.850811 14913 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:41.946751 14888 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.858057 14783 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:41.804508 14604 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:41.800379 14782 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.982316 12648 raft_consensus.cc:1275] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260202 15:08:42.086421 12648 pending_rounds.cc:85] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 2
I20260202 15:08:42.086772 12648 pending_rounds.cc:107] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260202 15:08:42.087057 12648 raft_consensus.cc:2889] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:41.980475 14564 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:41.985317 14902 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:41.985544 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:41.947127 14883 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.855122 14907 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
W20260202 15:08:41.988664 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:41.801756 12719 rpcz_store.cc:276] 0202 15:08:40.274841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.314875 (+ 40034us) service_pool.cc:225] Handling call
0202 15:08:40.330981 (+ 16106us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:40.331341 (+ 360us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:40.331371 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:40.331775 (+ 404us) log.cc:844] Serialized 26 byte log entry
0202 15:08:40.442229 (+110454us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:40.596344 (+154115us) log.cc:844] Serialized 14 byte log entry
0202 15:08:40.616908 (+ 20564us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:40.624469 (+ 7561us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:41.785759 (+1161290us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b54000666a0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4d4dee 00007f24ed4df960 00007f24ed4df908 00007f24ed4df888 00007f24ed4df843 00007f24ed4deacc 00007f24f3395a36 00007f24f33901e8 00007f24ed51b4fb 00007f24ed51dff0
0202 15:08:41.786311 (+ 552us) raft_consensus.cc:1685] finished
0202 15:08:41.786427 (+ 116us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:41.793086 (+ 6659us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":152669,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":6035,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":17212,"spinlock_wait_cycles":3317120,"thread_start_us":91842,"threads_started":1,"wal-append.queue_time_us":108123}
I20260202 15:08:41.946434 13356 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:41.843474 14535 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.138s
I20260202 15:08:41.855834 14870 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:41.993463 12765 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:41.988768 14673 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.845337 14904 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:41.962615 14854 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.013473 13361 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:41.850389 14850 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.986617 14908 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:41.991920 14176 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.964710 12793 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:41.994602 14511 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.017933 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:42.105209 12796 raft_consensus.cc:1275] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260202 15:08:42.121029 13363 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:42.101747 14468 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.102727 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:42.123215 12719 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260202 15:08:42.123919 12644 raft_consensus.cc:3060] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:42.124778 12719 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:08:42.031670 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:42.018338 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 184381us
I20260202 15:08:42.096729 14902 consensus_queue.cc:1048] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
I20260202 15:08:42.104427 14901 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.948474 14864 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:41.988903 12795 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:42.093408 12615 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:42.013415 14865 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.041460 14869 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:41.994230 14914 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.245s
I20260202 15:08:42.084105 14888 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:42.098839 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 105431us
I20260202 15:08:41.955227 14885 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:42.081218 13357 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:41.994644 14780 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:42.090170 14564 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:42.017613 14681 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.097379 13354 raft_consensus.cc:2410] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:42.090327 14913 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:42.100339 14535 consensus_queue.cc:1243] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:08:42.130017 14925 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.095975 14883 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.987149 14258 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:42.104628 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1461) took 1166 ms. Trace:
I20260202 15:08:42.104350 14896 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.094600 14907 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.133431 14176 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:42.128108 14915 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:42.185173 14915 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:42.185550 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 309) took 1227 ms. Trace:
I20260202 15:08:42.153486 14869 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.186160 12794 rpcz_store.cc:276] 0202 15:08:40.958168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.035300 (+ 77132us) service_pool.cc:225] Handling call
0202 15:08:41.047390 (+ 12090us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:41.139500 (+ 92110us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:41.139537 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:41.139584 (+ 47us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:41.139625 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:42.185271 (+1045646us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b54001f81d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:42.185334 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:42.185477 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":211404672}
I20260202 15:08:42.152985 14865 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.134611 14850 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:42.017374 12791 rpcz_store.cc:276] 0202 15:08:40.762568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.765216 (+ 2648us) service_pool.cc:225] Handling call
0202 15:08:40.788119 (+ 22903us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:40.824261 (+ 36142us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:40.826667 (+ 2406us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:40.868917 (+ 42250us) log.cc:844] Serialized 26 byte log entry
0202 15:08:41.017664 (+148747us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:41.051186 (+ 33522us) log.cc:844] Serialized 14 byte log entry
0202 15:08:41.090275 (+ 39089us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:41.095889 (+ 5614us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:41.850221 (+754332us) raft_consensus.cc:1685] finished
0202 15:08:41.850240 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:41.850359 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10392,"prepare.queue_time_us":26985,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":17950,"raft.queue_time_us":31796,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":15524,"thread_start_us":130491,"threads_started":1,"wal-append.queue_time_us":285282}
I20260202 15:08:42.122196 14468 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.137393 14673 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:42.136266 14845 consensus_meta.cc:287] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.792s user 0.010s sys 0.000s
I20260202 15:08:42.143793 14782 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:42.102874 14854 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:41.943270 13358 rpcz_store.cc:276] 0202 15:08:40.266606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.271421 (+ 4815us) service_pool.cc:225] Handling call
0202 15:08:41.452717 (+1181296us) spinlock_profiling.cc:243] Waited 5.86 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:41.452902 (+ 185us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:41.499156 (+ 46254us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:41.513161 (+ 14005us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:41.530933 (+ 17772us) log.cc:844] Serialized 26 byte log entry
0202 15:08:41.657085 (+126152us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:41.668813 (+ 11728us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:41.673176 (+ 4363us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:41.686060 (+ 12884us) raft_consensus.cc:1685] finished
0202 15:08:41.686087 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:41.731112 (+ 45025us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5888,"raft.queue_time_us":32038,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":14801,"spinlock_wait_cycles":13485824,"thread_start_us":77526,"threads_started":1,"wal-append.queue_time_us":109149}
I20260202 15:08:42.096424 14849 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.146018 14901 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.140914 14862 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.101711 14922 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:42.151000 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 57658.9us
I20260202 15:08:42.103231 14908 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:42.155228 14858 log.cc:927] Time spent T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 1.019s user 0.003s sys 0.000s
I20260202 15:08:42.128494 14889 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.162395 12689 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:42.153738 14902 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:42.159507 14564 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:42.176358 14925 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.162087 14913 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:42.185639 14915 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:42.133468 14872 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:42.366978 12644 raft_consensus.cc:1275] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:42.377998 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:42.345515 14918 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:42.378170 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1698) took 2119 ms. Trace:
I20260202 15:08:42.378743 14854 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:42.162988 13356 raft_consensus.cc:2410] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:42.379963 12723 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
W20260202 15:08:42.340593 14634 consensus_meta.cc:287] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.605s user 0.007s sys 0.002s
I20260202 15:08:42.273979 14864 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:42.527627 12792 raft_consensus.cc:1240] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:42.517694 12795 raft_consensus.cc:1275] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260202 15:08:42.277262 13354 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260202 15:08:42.381410 14862 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:42.507731 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1469) took 1073 ms. Trace:
I20260202 15:08:42.281174 14926 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.300335 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:42.537106 14926 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.538089 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:42.192516 14920 log.cc:927] Time spent T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260202 15:08:42.538627 12650 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:42.538879 14922 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:42.276890 14860 log.cc:927] Time spent T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 1.123s user 0.001s sys 0.000s
I20260202 15:08:42.540426 14931 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:42.174979 12793 rpcz_store.cc:276] 0202 15:08:40.938014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.939778 (+ 1764us) service_pool.cc:225] Handling call
0202 15:08:42.104525 (+1164747us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:42.378912 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1499) took 1318 ms. Trace:
I20260202 15:08:42.540184 14926 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:42.378583 12791 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:42.541182 14931 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.541875 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:42.266077 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:42.265992 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:42.482772 14787 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.873s user 0.013s sys 0.008s
I20260202 15:08:42.163262 14681 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.520804 14846 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:42.304481 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 1.10373e+06us
I20260202 15:08:42.340687 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:42.505738 14794 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.539559 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:42.406668 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 58969.4us
I20260202 15:08:42.470381 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:42.188050 12722 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:42.286492 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:42.541900 12644 rpcz_store.cc:276] 0202 15:08:41.060461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.060576 (+ 115us) service_pool.cc:225] Handling call
0202 15:08:41.091895 (+ 31319us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:42.123702 (+1031807us) spinlock_profiling.cc:243] Waited 995 us on lock 0x7b58005705d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4bb0da 00007f24ed53d177 00007f24ed555c67 00007f24ed55745e 00007f24ed5617c9 00007f24ed562096 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34
0202 15:08:42.378676 (+254974us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:42.378855 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2752640}
W20260202 15:08:42.535914 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 373560us
I20260202 15:08:42.273389 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 752) took 1339 ms. Trace:
I20260202 15:08:42.301004 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1789) took 1537 ms. Trace:
I20260202 15:08:42.297806 14896 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.553288 14933 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.381035 14865 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:42.752664 14872 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:42.732342 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 609) took 1179 ms. Trace:
I20260202 15:08:42.733409 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:42.267710 14832 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.378636 12721 rpcz_store.cc:276] 0202 15:08:40.258646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.258781 (+ 135us) service_pool.cc:225] Handling call
0202 15:08:40.275362 (+ 16581us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:40.423259 (+147897us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:40.423290 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:40.423680 (+ 390us) log.cc:844] Serialized 26 byte log entry
0202 15:08:41.019065 (+595385us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:41.019110 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:41.019163 (+ 53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:41.599006 (+579843us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b48001f0158. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f2478e21 00007f24f246feb7 00007f24f246fd4a 00007f24f2566f7f 00007f24f2566c47 00007f24f257b26c 00007f24f257b218 00007f24f257b198 00007f24f257b153 00007f24f257a3fc 00007f24f14b55b6
0202 15:08:42.154682 (+555676us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:42.377979 (+223297us) raft_consensus.cc:1685] finished
0202 15:08:42.377999 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:42.378102 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28422,"raft.queue_time_us":28705,"raft.run_cpu_time_us":2429,"raft.run_wall_time_us":1291937,"spinlock_wait_cycles":15103616,"thread_start_us":679261,"threads_started":2,"wal-append.queue_time_us":617325}
I20260202 15:08:42.755193 12650 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:42.554190 12615 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:42.275513 14916 log.cc:927] Time spent T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.187s user 0.001s sys 0.001s
W20260202 15:08:42.452069 14802 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260202 15:08:42.764674 14846 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.795126 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:42.795063 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:42.796104 12721 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:42.796459 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:42.796653 14846 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:42.797047 12794 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:42.797648 14850 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:42.797784 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:42.798313 12794 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:42.798133 14850 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.799070 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:42.759714 12765 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:42.717252 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 610) took 1148 ms. Trace:
I20260202 15:08:42.799491 14922 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:42.799626 12794 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
W20260202 15:08:42.758049 14787 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.151s user 0.020s sys 0.014s
I20260202 15:08:42.757886 13363 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:42.761008 12723 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:42.762552 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:42.756774 12614 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:42.190399 14883 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:42.803956 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 44327.6us
I20260202 15:08:42.791633 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 336) took 1001 ms. Trace:
I20260202 15:08:42.187829 14869 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:42.438932 14873 raft_consensus.cc:2749] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:42.267088 14511 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:42.465432 14888 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:42.469630 14834 log.cc:927] Time spent T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 1.710s user 0.000s sys 0.003s
I20260202 15:08:42.766527 14794 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:42.799932 14850 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:42.379491 14907 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:42.800938 12644 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:42.362146 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 553) took 1664 ms. Trace:
W20260202 15:08:42.801357 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 44658.8us
I20260202 15:08:42.437759 14913 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
W20260202 15:08:42.777060 12764 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.46.196:36191 timed out after 2.162s (SENT) [suppressed 47 similar messages]
W20260202 15:08:42.794495 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 240372us
I20260202 15:08:42.342281 14176 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:42.799800 13356 rpcz_store.cc:276] 0202 15:08:41.568625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.517402 (+948777us) service_pool.cc:225] Handling call
0202 15:08:42.533771 (+ 16369us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:42.554345 (+ 20574us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:08:42.585994 (+ 31649us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:42.638340 (+ 52346us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:42.642222 (+ 3882us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:42.663436 (+ 21214us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:42.717178 (+ 53742us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:42.756176 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:42.378656 14918 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.284910 14258 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.454648 14845 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.052060 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:43.091157 12615 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:42.275547 14927 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.717425 14722 raft_consensus.cc:3055] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:42.658277 14870 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.552544 14902 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:42.553794 14634 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.657614 14930 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:42.758950 14604 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:42.627697 14764 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.133615 12644 pending_rounds.cc:85] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 5
I20260202 15:08:42.876708 12721 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:43.137854 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:43.092097 12615 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:42.875978 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:43.138752 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 46722.1us
I20260202 15:08:43.138996 12794 raft_consensus.cc:3055] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:43.090411 12641 rpcz_store.cc:276] 0202 15:08:41.789832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.340348 (+550516us) service_pool.cc:225] Handling call
0202 15:08:42.791567 (+451219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:43.085599 12723 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:43.139927 12615 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:43.141093 12644 pending_rounds.cc:107] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260202 15:08:43.141359 12644 raft_consensus.cc:2889] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:43.113787 12689 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:42.579843 14854 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:42.836575 12724 raft_consensus.cc:2393] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:42.757647 12797 rpcz_store.cc:276] 0202 15:08:40.763234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.771133 (+ 7899us) service_pool.cc:225] Handling call
0202 15:08:40.777910 (+ 6777us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:40.821983 (+ 44073us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:42.300682 (+1478699us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b54002f1d20. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4d6df0 00007f24ed562682 00007f24ed55ea56 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22
0202 15:08:42.300938 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3377920}
I20260202 15:08:43.135459 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1427) took 1782 ms. Trace:
I20260202 15:08:43.139338 12794 raft_consensus.cc:740] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:08:42.981365 14890 log.cc:927] Time spent T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 1.634s user 0.001s sys 0.001s
I20260202 15:08:43.156107 12726 rpcz_store.cc:276] 0202 15:08:41.353259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.461400 (+108141us) service_pool.cc:225] Handling call
0202 15:08:41.517599 (+ 56199us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:41.525889 (+ 8290us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:41.525922 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:41.557495 (+ 31573us) log.cc:844] Serialized 26 byte log entry
0202 15:08:41.800633 (+243138us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:41.800665 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:41.800722 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:42.274833 (+474111us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:43.096746 (+821913us) raft_consensus.cc:1685] finished
0202 15:08:43.096777 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.135377 (+ 38600us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11732,"raft.queue_time_us":7690,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":19588,"spinlock_wait_cycles":5885184,"thread_start_us":218794,"threads_started":1,"wal-append.queue_time_us":380033}
I20260202 15:08:42.847278 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:43.158840 12650 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:43.163756 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:43.074546 14673 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.163241 12794 consensus_queue.cc:260] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.792604 14847 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:43.070868 14841 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:42.762667 14872 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.096943 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:42.996176 14901 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.104040 14468 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:42.888195 14925 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:42.806329 14856 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:08:42.873890 14857 consensus_queue.cc:1059] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.403s
I20260202 15:08:43.232764 14883 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:43.101639 12720 raft_consensus.cc:1240] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:42.858451 14926 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:43.113778 14917 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.834896 14787 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:42.800832 12793 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:42.836441 14865 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.105018 14802 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:43.103164 14869 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:42.803965 14849 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:42.803745 14502 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:43.166322 14764 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.554816 14889 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:42.886183 14931 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.096369 14884 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:43.235754 12800 raft_consensus.cc:2393] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:43.130968 14902 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:42.755009 14933 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:42.764846 14564 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.134684 14722 raft_consensus.cc:740] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:42.794109 14832 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:43.250641 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1465) took 2296 ms. Trace:
I20260202 15:08:43.251848 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:43.242928 14841 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.252645 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:43.230024 12794 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:43.235827 14885 consensus_meta.cc:287] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.078s user 0.011s sys 0.000s
I20260202 15:08:43.141952 14927 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.206820 14907 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:43.300174 14933 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.300397 14907 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:43.300745 14907 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:43.276784 14869 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.301646 14927 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.302268 14869 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.167776 12718 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260202 15:08:43.237919 14873 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:43.166688 14604 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:43.332397 14845 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.333900 14604 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.137735 14870 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.128695 14511 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:43.266564 13357 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.005s sys 0.017s
W20260202 15:08:43.276111 13362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 551) took 2639 ms (client timeout 2162 ms). Trace:
W20260202 15:08:43.341271 13362 rpcz_store.cc:269] 0202 15:08:40.633813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.633949 (+ 136us) service_pool.cc:225] Handling call
0202 15:08:43.272584 (+2638635us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400470254. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:43.272676 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":370622336}
I20260202 15:08:43.262898 14856 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:43.249249 14867 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260202 15:08:43.270224 14883 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.280073 12765 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:43.350492 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 70480.2us
I20260202 15:08:43.287214 14884 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.265770 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:43.352589 12797 raft_consensus.cc:2393] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:43.352789 14884 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.353655 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:43.164201 14930 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.357671 14930 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.232682 14871 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260202 15:08:43.234629 14847 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.361124 14847 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.361670 14847 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.270776 14857 consensus_queue.cc:1243] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 1eca64246cad41379cde4e2f9c3e25d2 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260202 15:08:43.269050 14926 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:43.363308 14926 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:43.278489 14681 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.101341 14176 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:43.366771 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 274535us
I20260202 15:08:43.368510 13328 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:43.368860 14176 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.369524 14176 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:43.370031 14176 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.235507 14673 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.290006 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1498) took 2121 ms. Trace:
W20260202 15:08:43.376171 14908 consensus_meta.cc:287] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.786s user 0.008s sys 0.000s
I20260202 15:08:43.376641 14908 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.376358 12643 rpcz_store.cc:276] 0202 15:08:41.168612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.171036 (+ 2424us) service_pool.cc:225] Handling call
0202 15:08:42.146206 (+975170us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:42.146410 (+ 204us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:42.188810 (+ 42400us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:42.188847 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:42.188896 (+ 49us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:42.188956 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.289763 (+1100807us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b54002a9754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:43.289813 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.289937 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":102526592}
I20260202 15:08:43.377533 14673 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.290051 14564 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.290524 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:43.291288 14932 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.220s user 0.000s sys 0.000s
I20260202 15:08:43.292335 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:43.287232 12796 rpcz_store.cc:276] 0202 15:08:40.942062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.957160 (+ 15098us) service_pool.cc:225] Handling call
0202 15:08:42.104740 (+1147580us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:42.118817 (+ 14077us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.238277 (+1119460us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400116f50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:43.238386 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2582758272}
I20260202 15:08:43.294037 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:43.299484 14872 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:43.299809 14849 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:43.273124 14896 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.270980 14835 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:42.760488 13354 rpcz_store.cc:276] 0202 15:08:41.553284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.176874 (+623590us) service_pool.cc:225] Handling call
0202 15:08:42.267574 (+ 90700us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:42.578813 (+311239us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:42.732150 (+153337us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b54006ade50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:42.732287 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36551936}
I20260202 15:08:43.134323 13355 rpcz_store.cc:276] 0202 15:08:40.698179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.731518 (+ 33339us) service_pool.cc:225] Handling call
0202 15:08:40.794520 (+ 63002us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:41.163144 (+368624us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:42.361900 (+1198756us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54004d9c54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:42.362082 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":658608512}
I20260202 15:08:43.272755 14802 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:42.755740 13359 rpcz_store.cc:276] 0202 15:08:40.933992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.936270 (+ 2278us) service_pool.cc:225] Handling call
0202 15:08:41.008152 (+ 71882us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:41.079557 (+ 71405us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:41.085427 (+ 5870us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:41.174503 (+ 89076us) log.cc:844] Serialized 52 byte log entry
0202 15:08:42.190943 (+1016440us) spinlock_profiling.cc:243] Waited 76.8 ms on lock 0x7b18003b9b00. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ed10685c 00007f24ea4942eb 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471
0202 15:08:42.191127 (+ 184us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:42.191174 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:42.191259 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:42.191310 (+ 51us) raft_consensus.cc:1685] finished
0202 15:08:42.191329 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:42.273322 (+ 81993us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4899,"raft.queue_time_us":20335,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":29741,"spinlock_wait_cycles":176738688,"thread_start_us":142972,"threads_started":1,"wal-append.queue_time_us":147829,"wal-append.run_cpu_time_us":3360,"wal-append.run_wall_time_us":1363913}
I20260202 15:08:43.381093 14846 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:43.340869 14913 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:43.326411 14935 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.327009 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:43.327512 14764 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:43.328678 14832 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.330349 14722 consensus_queue.cc:260] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:43.272469 14917 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.331867 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1472) took 1680 ms. Trace:
I20260202 15:08:43.331914 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1784) took 3072 ms. Trace:
I20260202 15:08:43.335044 12723 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:43.335043 12721 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:43.302881 14869 raft_consensus.cc:2804] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:43.403227 14258 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.276675 14787 raft_consensus.cc:697] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:43.169255 14854 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:43.342973 14915 consensus_meta.cc:287] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.832s user 0.004s sys 0.004s
I20260202 15:08:42.545339 12798 rpcz_store.cc:276] 0202 15:08:41.434577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.964551 (+529974us) service_pool.cc:225] Handling call
0202 15:08:41.982436 (+ 17885us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:42.093446 (+111010us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:42.093476 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:42.094158 (+ 682us) log.cc:844] Serialized 26 byte log entry
0202 15:08:42.408385 (+314227us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:42.506905 (+ 98520us) log.cc:844] Serialized 14 byte log entry
0202 15:08:42.507297 (+ 392us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:42.507445 (+ 148us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:42.507497 (+ 52us) raft_consensus.cc:1685] finished
0202 15:08:42.507516 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:42.507662 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11078,"prepare.queue_time_us":98262,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":136877,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":15610,"spinlock_wait_cycles":1330944,"thread_start_us":407622,"threads_started":2,"wal-append.queue_time_us":282000}
I20260202 15:08:43.268760 14936 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:43.419258 14940 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:43.346045 13357 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.982s user 0.005s sys 0.017s
I20260202 15:08:43.371621 14868 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
W20260202 15:08:43.380448 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 266700us
I20260202 15:08:43.439962 12689 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:43.331885 14841 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.380486 12646 raft_consensus.cc:2368] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:43.354408 12797 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:43.362151 14847 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.391625 13362 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEARNER]: Advancing to term 4
I20260202 15:08:43.348541 14856 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:08:43.280192 14502 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.348217 14931 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:43.303016 14794 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:43.325759 12791 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:43.277244 14865 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:43.406719 14802 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:43.403791 14634 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.416507 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1471) took 1826 ms. Trace:
I20260202 15:08:43.401895 14939 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.419837 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:43.307541 14885 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:43.403551 14918 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.419870 12800 rpcz_store.cc:276] 0202 15:08:41.651226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.956486 (+305260us) service_pool.cc:225] Handling call
0202 15:08:43.331699 (+1375213us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54001cd4d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:43.331804 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2855804288}
I20260202 15:08:43.371865 14511 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.346119 14873 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.345672 12794 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:43.404784 14835 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.479857 14681 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.402336 12719 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:43.419991 12799 rpcz_store.cc:276] 0202 15:08:40.259833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.259958 (+ 125us) service_pool.cc:225] Handling call
0202 15:08:40.261927 (+ 1969us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:40.262281 (+ 354us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:40.262309 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:40.262895 (+ 586us) log.cc:844] Serialized 52 byte log entry
0202 15:08:40.591414 (+328519us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:40.618376 (+ 26962us) log.cc:844] Serialized 14 byte log entry
0202 15:08:40.618642 (+ 266us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:40.618737 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:42.469316 (+1850579us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b240011bc88. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea305ad4 00007f24ea303bdb 00007f24ea392ad9 00007f24ea392765 00007f24ea391dad 00007f24ed0e24ae 00007f24ed0f9c2c 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1
0202 15:08:42.578948 (+109632us) spinlock_profiling.cc:243] Waited 9.34 ms on lock 0x7b1800390280. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea4b12d3 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad 0000000000379254 0000000000379099 00007f24ea49469a
0202 15:08:43.331711 (+752763us) raft_consensus.cc:1685] finished
0202 15:08:43.331732 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.331847 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22985,"prepare.queue_time_us":29719,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":5246,"raft.queue_time_us":3538,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":152,"spinlock_wait_cycles":48960256,"thread_start_us":328037,"threads_started":1,"wal-append.queue_time_us":422060}
I20260202 15:08:43.509644 12798 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:08:43.428004 14908 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.410481 14888 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:43.430195 14176 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:43.430374 14913 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:43.419188 14935 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.431457 14787 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.423970 14846 raft_consensus.cc:697] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:43.423456 14869 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:43.378209 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 601148us
I20260202 15:08:43.420413 12793 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
W20260202 15:08:43.422295 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 304) took 2458 ms (client timeout 1821 ms). Trace:
W20260202 15:08:43.443382 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 114022us
I20260202 15:08:43.436451 14535 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:43.479698 14868 raft_consensus.cc:697] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:43.433197 14915 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:43.474025 14258 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:43.484359 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 612) took 1847 ms. Trace:
I20260202 15:08:43.511377 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1797) took 1350 ms. Trace:
I20260202 15:08:43.512216 12765 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:43.491118 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1708) took 2290 ms. Trace:
I20260202 15:08:43.491775 12647 raft_consensus.cc:3055] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:43.496871 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 611) took 1912 ms. Trace:
I20260202 15:08:43.498291 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:43.499055 12796 rpcz_store.cc:276] 0202 15:08:41.589983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.400350 (+1810367us) service_pool.cc:225] Handling call
0202 15:08:43.416046 (+ 15696us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:43.416216 (+ 170us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:43.416239 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:43.416278 (+ 39us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:43.416311 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.416360 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.416458 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:43.499298 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:43.504926 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1502) took 2065 ms. Trace:
I20260202 15:08:43.505936 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1431) took 1940 ms. Trace:
I20260202 15:08:43.521417 13362 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260202 15:08:43.512679 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 401531us
I20260202 15:08:43.440050 14722 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:43.436674 14936 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.443709 14940 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:43.491984 14847 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.223501 14832 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.499321 14939 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.948622 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1788) took 3185 ms. Trace:
I20260202 15:08:44.230417 14832 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.230660 12792 rpcz_store.cc:276] 0202 15:08:40.762934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.765254 (+ 2320us) service_pool.cc:225] Handling call
0202 15:08:41.984478 (+1219224us) spinlock_profiling.cc:243] Waited 8.59 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea47412e 000000000035a3c9 00007f24f249e560 00007f24f2566dd4 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:41.984614 (+ 136us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:42.535698 (+551084us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.948390 (+1412692us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b5400371ed4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:43.948561 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1255527424}
I20260202 15:08:44.231611 12614 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:44.232132 14939 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.510214 14854 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:44.232529 14832 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.486483 14889 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.508268 14873 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.232950 12792 raft_consensus.cc:1275] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:43.545406 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1442) took 1264 ms. Trace:
I20260202 15:08:44.234040 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1475) took 1890 ms. Trace:
I20260202 15:08:43.574983 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1496) took 2869 ms. Trace:
W20260202 15:08:43.935791 12722 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1720) took 1949 ms (client timeout 2056 ms). Trace:
I20260202 15:08:43.539987 14926 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:44.262452 12722 rpcz_store.cc:269] 0202 15:08:41.981768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.017743 (+ 35975us) service_pool.cc:225] Handling call
0202 15:08:43.931408 (+1913665us) spinlock_profiling.cc:243] Waited 55.6 ms on lock 0x7b5400157554. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:43.931517 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127964032}
I20260202 15:08:44.235761 14935 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.495412 14938 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:43.542157 14634 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:44.236442 12647 raft_consensus.cc:740] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:43.666013 14918 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.935312 14950 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
W20260202 15:08:44.237512 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 725326us
I20260202 15:08:44.245659 14915 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:44.293112 12765 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:43.508306 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 558) took 1989 ms. Trace:
I20260202 15:08:43.905813 14793 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.238135 14940 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:43.905252 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 563) took 1377 ms. Trace:
I20260202 15:08:43.666432 14835 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.994809 14930 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.238914 14258 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
W20260202 15:08:43.950457 12724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1721) took 1964 ms (client timeout 1629 ms). Trace:
I20260202 15:08:43.880643 14511 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:43.751020 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:43.676551 14802 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:44.240558 12717 rpcz_store.cc:269] 0202 15:08:40.958784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.975329 (+ 16545us) service_pool.cc:225] Handling call
0202 15:08:43.283903 (+2308574us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b58001d7098. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:43.410127 (+126224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2426752}
I20260202 15:08:44.289628 14889 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:43.826176 12763 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:44.262077 12792 rpcz_store.cc:276] 0202 15:08:42.343432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.231975 (+1888543us) service_pool.cc:225] Handling call
0202 15:08:44.232414 (+ 439us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:44.233745 (+ 1331us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.233962 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:43.479912 14896 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.241339 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1476) took 1897 ms. Trace:
I20260202 15:08:43.678949 14902 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.241711 12642 rpcz_store.cc:276] 0202 15:08:41.440011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.440159 (+ 148us) service_pool.cc:225] Handling call
0202 15:08:43.504730 (+2064571us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b54001a1dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:43.504856 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":134148352}
I20260202 15:08:43.494947 14931 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:43.634781 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 405141us
I20260202 15:08:43.552208 14681 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.525158 14885 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:43.501226 12797 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260202 15:08:44.243522 14535 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:43.680774 14945 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:44.314145 14258 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:44.244660 12800 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:44.244854 13362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 616) took 2114 ms. Trace:
I20260202 15:08:44.345259 13362 rpcz_store.cc:276] 0202 15:08:42.129786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.372528 (+1242742us) service_pool.cc:225] Handling call
0202 15:08:43.391483 (+ 18955us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:44.244276 (+852793us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.244458 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:43.908198 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 337) took 1622 ms (client timeout 1716 ms). Trace:
I20260202 15:08:43.496743 14856 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:43.490126 14849 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:44.314549 14835 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:44.346101 12646 rpcz_store.cc:269] 0202 15:08:42.286109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.286248 (+ 139us) service_pool.cc:225] Handling call
0202 15:08:43.908014 (+1621766us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b540024f754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:43.908136 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1427518720}
I20260202 15:08:44.245910 12721 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
W20260202 15:08:43.966029 12791 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1804) took 1428 ms (client timeout 1629 ms). Trace:
I20260202 15:08:43.951380 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 559) took 2319 ms. Trace:
I20260202 15:08:44.246879 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1477) took 1706 ms. Trace:
I20260202 15:08:43.891510 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1504) took 2229 ms. Trace:
I20260202 15:08:44.247400 13363 rpcz_store.cc:276] 0202 15:08:41.636832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.800668 (+1163836us) service_pool.cc:225] Handling call
0202 15:08:42.836152 (+ 35484us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:43.370930 (+534778us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:43.370969 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:43.371020 (+ 51us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:43.371069 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.484165 (+113096us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b54005eddd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:43.484203 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.484308 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1229324160}
I20260202 15:08:43.490666 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1711) took 1966 ms. Trace:
I20260202 15:08:44.248242 12723 raft_consensus.cc:2393] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:44.229626 12798 rpcz_store.cc:276] 0202 15:08:42.160980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.434007 (+1273027us) service_pool.cc:225] Handling call
0202 15:08:43.473216 (+ 39209us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:43.511167 (+ 37951us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.511331 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:43.931355 14913 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.229362 14865 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.229858 14869 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:44.248929 13358 rpcz_store.cc:276] 0202 15:08:41.584366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.496043 (+1911677us) service_pool.cc:225] Handling call
0202 15:08:43.496364 (+ 321us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:43.496576 (+ 212us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:43.496605 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:43.496644 (+ 39us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:43.496675 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.496727 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.496823 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:43.949208 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 345) took 1601 ms. Trace:
I20260202 15:08:44.238636 14846 consensus_queue.cc:237] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:43.649273 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 280830us
I20260202 15:08:43.621639 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 597) took 2832 ms. Trace:
I20260202 15:08:44.013669 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.240020 14936 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:43.495922 13355 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:44.392987 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.393327 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 761) took 1637 ms. Trace:
I20260202 15:08:44.393700 13355 rpcz_store.cc:276] 0202 15:08:42.755727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.425224 (+669497us) service_pool.cc:225] Handling call
0202 15:08:43.495601 (+ 70377us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:44.393066 (+897465us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.393272 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.388944 13360 rpcz_store.cc:276] 0202 15:08:40.711906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.712061 (+ 155us) service_pool.cc:225] Handling call
0202 15:08:40.712314 (+ 253us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:40.753047 (+ 40733us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:40.753084 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:40.771434 (+ 18350us) log.cc:844] Serialized 26 byte log entry
0202 15:08:43.237536 (+2466102us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b18000bd9a0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471 00007f24ed51a73f
0202 15:08:43.436294 (+198758us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:43.436332 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.436642 (+ 310us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:43.516725 (+ 80083us) raft_consensus.cc:1685] finished
0202 15:08:43.516755 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.544636 (+ 27881us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":77852,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":6728,"spinlock_wait_cycles":2778368,"thread_start_us":198252,"threads_started":1,"wal-append.queue_time_us":2706529}
I20260202 15:08:44.394804 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.394843 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.395648 14931 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.396235 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.397464 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.233984 12719 rpcz_store.cc:276] 0202 15:08:42.280780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.293809 (+1013029us) service_pool.cc:225] Handling call
0202 15:08:43.540375 (+246566us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.398044 13359 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:44.398613 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:44.398947 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.399174 12719 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:44.399523 13359 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:44.400478 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:43.551606 12799 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260202 15:08:43.648325 14841 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:43.524380 14917 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.355638 14945 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.389547 14913 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.263275 12645 rpcz_store.cc:276] 0202 15:08:40.705463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.705652 (+ 189us) service_pool.cc:225] Handling call
0202 15:08:40.706024 (+ 372us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:41.796129 (+1090105us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.574762 (+1778633us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540029a754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:43.574935 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2398278912}
W20260202 15:08:44.251407 14762 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.004s sys 0.011s
W20260202 15:08:44.236728 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:43.622339 14927 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:44.290150 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.384503 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1799) took 2217 ms. Trace:
I20260202 15:08:44.296823 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.301231 14793 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.295193 14918 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:44.309499 12724 rpcz_store.cc:269] 0202 15:08:41.986482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.991652 (+ 5170us) service_pool.cc:225] Handling call
0202 15:08:43.950276 (+1958624us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b540033b154. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:43.950393 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1578246912}
W20260202 15:08:43.888008 14907 consensus_meta.cc:287] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.587s user 0.008s sys 0.000s
I20260202 15:08:44.318284 12796 rpcz_store.cc:276] 0202 15:08:42.343741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.240664 (+1896923us) service_pool.cc:225] Handling call
0202 15:08:44.240929 (+ 265us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:44.241070 (+ 141us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:44.241091 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:44.241124 (+ 33us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:44.241152 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.241194 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:44.241270 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:43.633874 14851 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:44.237639 14893 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:44.232386 14952 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:44.261911 14924 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.242471 12725 rpcz_store.cc:276] 0202 15:08:41.200438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.207100 (+ 6662us) service_pool.cc:225] Handling call
0202 15:08:41.207388 (+ 288us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:41.207691 (+ 303us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:41.207713 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:42.184260 (+976547us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b2c00010828. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ebe5d9b1 00007f24ebe6cfe2 00007f24ed59a851 00007f24ed55f0ba 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22
0202 15:08:42.184880 (+ 620us) log.cc:844] Serialized 52 byte log entry
0202 15:08:42.640987 (+456107us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:42.929779 (+288792us) log.cc:844] Serialized 14 byte log entry
0202 15:08:43.268581 (+338802us) log.cc:844] Serialized 14 byte log entry
0202 15:08:43.269560 (+ 979us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.269627 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:43.490931 (+221304us) raft_consensus.cc:1685] finished
0202 15:08:43.490956 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.491068 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61097,"prepare.queue_time_us":640560,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":313087,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":104732,"spinlock_wait_cycles":14758656,"thread_start_us":1018643,"threads_started":3,"wal-append.queue_time_us":701497}
W20260202 15:08:44.345467 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 410510us
I20260202 15:08:44.415532 14913 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.298779 14950 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.357695 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.309122 14930 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.234220 14868 consensus_queue.cc:237] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.294476 14634 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:44.322867 14511 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:44.357041 13361 rpcz_store.cc:276] 0202 15:08:41.632242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.730805 (+1098563us) service_pool.cc:225] Handling call
0202 15:08:42.731068 (+ 263us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:42.802091 (+ 71023us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:08:43.266081 (+463990us) log.cc:844] Serialized 14 byte log entry
0202 15:08:43.518523 (+252442us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:43.518569 (+ 46us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:43.518606 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.951181 (+432575us) spinlock_profiling.cc:243] Waited 70.7 ms on lock 0x7b54004dc6d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:43.951228 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.951333 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92378,"prepare.queue_time_us":575703,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":212400,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":34589,"spinlock_wait_cycles":164338304,"thread_start_us":351391,"threads_started":2,"wal-append.queue_time_us":400065}
I20260202 15:08:44.504240 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.382525 12726 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:44.358907 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1452) took 1003 ms. Trace:
I20260202 15:08:44.291018 14938 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:44.384295 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 91278.8us
I20260202 15:08:44.505509 12721 rpcz_store.cc:276] 0202 15:08:43.355608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.499054 (+143446us) service_pool.cc:225] Handling call
0202 15:08:44.358828 (+859774us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:44.506868 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:44.507663 12765 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:44.359112 12648 rpcz_store.cc:276] 0202 15:08:41.661750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.661858 (+ 108us) service_pool.cc:225] Handling call
0202 15:08:41.679702 (+ 17844us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:42.090269 (+410567us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.891242 (+1800973us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54000fa0d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:43.891447 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":655894912}
I20260202 15:08:44.508343 12765 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:44.385816 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.509650 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.510757 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:44.511253 13363 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:44.389346 13328 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:44.504978 13361 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:08:44.513434 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 124143us
I20260202 15:08:44.514178 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.388710 12644 rpcz_store.cc:276] 0202 15:08:42.347442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.114405 (+766963us) service_pool.cc:225] Handling call
0202 15:08:43.133354 (+ 18949us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:43.163086 (+ 29732us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:43.163118 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:43.163655 (+ 537us) log.cc:844] Serialized 52 byte log entry
0202 15:08:43.559664 (+396009us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:43.559787 (+ 123us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.559868 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:43.948966 (+389098us) raft_consensus.cc:1685] finished
0202 15:08:43.948994 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.949143 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21175,"raft.queue_time_us":21953,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":61358,"thread_start_us":296267,"threads_started":1,"wal-append.queue_time_us":698614}
I20260202 15:08:44.356658 14849 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:44.403187 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.411948 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1801) took 2144 ms. Trace:
I20260202 15:08:44.329288 14889 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.403745 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.412181 14841 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.381449 14835 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.328754 14896 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.427547 12649 raft_consensus.cc:2393] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
W20260202 15:08:44.557056 12649 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1505) took 2847 ms (client timeout 3347 ms). Trace:
I20260202 15:08:44.503388 14511 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:44.424244 12689 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:44.559854 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 135699us
I20260202 15:08:44.560503 12689 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:44.449529 12722 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
W20260202 15:08:44.422123 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:44.561354 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1450) took 1274 ms. Trace:
I20260202 15:08:44.412441 14917 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.245151 12718 rpcz_store.cc:276] 0202 15:08:41.565168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.799258 (+234090us) service_pool.cc:225] Handling call
0202 15:08:41.799572 (+ 314us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:43.302992 (+1503420us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.505707 (+202715us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400031e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:43.505862 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3145986944}
I20260202 15:08:44.563512 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.296368 13357 rpcz_store.cc:276] 0202 15:08:41.518835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.168689 (+649854us) service_pool.cc:225] Handling call
0202 15:08:42.362245 (+193556us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:42.363173 (+ 928us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:42.363219 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:42.363853 (+ 634us) log.cc:844] Serialized 26 byte log entry
0202 15:08:43.473643 (+1109790us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:43.507857 (+ 34214us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.507981 (+ 124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:43.508055 (+ 74us) raft_consensus.cc:1685] finished
0202 15:08:43.508073 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.508234 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":116,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":1072326,"threads_started":1,"wal-append.queue_time_us":1012462,"wal-append.run_cpu_time_us":3921,"wal-append.run_wall_time_us":1008087}
I20260202 15:08:44.458762 14893 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:44.473656 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.458796 14952 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:44.358994 12793 rpcz_store.cc:276] 0202 15:08:42.540254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.287483 (+747229us) service_pool.cc:225] Handling call
0202 15:08:44.246814 (+959331us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.498183 14924 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.456825 14907 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:44.483253 14964 log.cc:927] Time spent T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
W20260202 15:08:44.484012 14965 log.cc:927] Time spent T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260202 15:08:44.471565 14969 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:44.477288 14972 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.473003 14793 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.485199 14918 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.393024 14971 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:44.415584 14945 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.485589 13360 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 2.
I20260202 15:08:44.403559 14915 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.461951 13326 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:44.487521 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.475032 14926 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:44.479847 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1470) took 2891 ms. Trace:
I20260202 15:08:44.476392 14902 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.479235 14883 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:44.420177 14762 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s user 0.004s sys 0.011s
I20260202 15:08:44.459056 12796 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:44.604389 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 319) took 1730 ms. Trace:
I20260202 15:08:44.398051 13355 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:44.453106 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.497517 13362 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
W20260202 15:08:44.357110 12791 rpcz_store.cc:269] 0202 15:08:42.537258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.659656 (+122398us) service_pool.cc:225] Handling call
0202 15:08:43.965808 (+1306152us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54001f86d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:43.965945 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":466773760}
I20260202 15:08:44.427083 14931 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260202 15:08:44.520303 13361 raft_consensus.cc:2368] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4.
I20260202 15:08:44.547736 12723 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:44.624389 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1451) took 1269 ms. Trace:
I20260202 15:08:44.413616 12798 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260202 15:08:44.547847 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.520810 12799 rpcz_store.cc:276] 0202 15:08:42.267962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.511217 (+1243255us) service_pool.cc:225] Handling call
0202 15:08:43.551352 (+ 40135us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:44.403265 (+851913us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.411892 (+ 8627us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.530903 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.625306 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 316) took 1860 ms. Trace:
I20260202 15:08:44.530501 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.520838 13358 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 1.
I20260202 15:08:44.451437 14927 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.553680 13363 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:44.383896 12720 rpcz_store.cc:276] 0202 15:08:41.524053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.695759 (+171706us) service_pool.cc:225] Handling call
0202 15:08:43.101418 (+1405659us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:43.271333 (+169915us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.490486 (+219153us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400159350. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:43.490609 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3035849984}
W20260202 15:08:44.557463 12649 rpcz_store.cc:269] 0202 15:08:41.709183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.253749 (+2544566us) service_pool.cc:225] Handling call
0202 15:08:44.556997 (+303248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.289458 12647 consensus_queue.cc:260] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.561743 12722 rpcz_store.cc:276] 0202 15:08:43.288699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.296511 (+1007812us) service_pool.cc:225] Handling call
0202 15:08:44.561269 (+264758us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.594331 14976 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.449370 12797 rpcz_store.cc:276] 0202 15:08:42.167052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.500706 (+1333654us) service_pool.cc:225] Handling call
0202 15:08:43.500957 (+ 251us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:44.342929 (+841972us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.382882 (+ 39953us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.672945 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.547364 14913 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:44.564059 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:44.521776 14849 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:44.567138 12725 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:44.569242 12689 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:44.547571 14885 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:08:44.464892 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 143717us
I20260202 15:08:44.570253 14936 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:44.575217 12764 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:44.576531 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:44.521590 14962 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:44.563814 14511 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:44.569414 14893 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:44.570398 14865 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:44.569571 14802 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.590297 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.569736 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.522855 14258 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.569156 14952 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:44.596117 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 134257us
I20260202 15:08:44.596316 12724 raft_consensus.cc:2393] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:44.594905 12615 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:44.596550 12795 rpcz_store.cc:276] 0202 15:08:41.588763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:42.149426 (+560663us) service_pool.cc:225] Handling call
0202 15:08:42.149720 (+ 294us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:42.535899 (+386179us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.479615 (+1943716us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b54001f95d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:44.479768 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":829975936}
I20260202 15:08:44.600396 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.702921 14865 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:44.703104 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:44.602785 14851 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:44.703392 13357 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:44.703706 14851 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:44.704248 13326 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:44.705646 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:44.569691 14917 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.612185 14938 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.707464 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.613433 14889 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.590853 14856 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:44.592460 14942 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.017s sys 0.017s
W20260202 15:08:44.708385 12726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1453) took 1326 ms (client timeout 1716 ms). Trace:
W20260202 15:08:44.708698 14942 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s user 0.017s sys 0.017s
W20260202 15:08:44.708837 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 139659us
I20260202 15:08:44.625607 12798 rpcz_store.cc:276] 0202 15:08:42.765327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.387058 (+1621731us) service_pool.cc:225] Handling call
0202 15:08:44.413345 (+ 26287us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:44.625110 (+211765us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.625263 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.627166 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.595948 14844 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.632495 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:44.557294 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.633592 12765 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:44.614001 14935 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:44.650257 14924 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.629112 14722 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.654273 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.654398 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.623950 14902 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:44.667351 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.587918 14969 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.596364 14971 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.667976 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.668700 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.590051 14972 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.604794 14915 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.594712 14793 consensus_queue.cc:1059] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260202 15:08:44.605281 14869 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.695639 14927 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:44.606346 14970 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.614779 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.624739 12723 rpcz_store.cc:276] 0202 15:08:43.355243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.385635 (+1030392us) service_pool.cc:225] Handling call
0202 15:08:44.624337 (+238702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.687598 12647 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:44.807619 12645 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
W20260202 15:08:44.700397 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 105577us
I20260202 15:08:44.700841 14973 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:44.701180 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 311) took 1158 ms. Trace:
I20260202 15:08:44.601316 14835 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.604808 12796 rpcz_store.cc:276] 0202 15:08:42.874333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.454381 (+1580048us) service_pool.cc:225] Handling call
0202 15:08:44.454612 (+ 231us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:44.604110 (+149498us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.604337 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.705786 14918 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:44.817108 14918 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:44.817468 14918 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:44.819922 14959 log.cc:927] Time spent T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.496s user 0.001s sys 0.002s
I20260202 15:08:44.821022 14917 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:44.706535 14634 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.589389 14681 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.708019 12795 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:44.622151 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1474) took 2279 ms. Trace:
I20260202 15:08:44.828197 14258 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.827365 12794 rpcz_store.cc:276] 0202 15:08:42.343085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.540803 (+1197718us) service_pool.cc:225] Handling call
0202 15:08:43.541153 (+ 350us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:43.541535 (+ 382us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:43.541561 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:43.622110 (+ 80549us) log.cc:844] Serialized 26 byte log entry
0202 15:08:44.309390 (+687280us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:44.309427 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.309562 (+ 135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:44.621961 (+312399us) raft_consensus.cc:1685] finished
0202 15:08:44.621981 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:44.622087 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":176487,"raft.queue_time_us":648,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":88175,"thread_start_us":483730,"threads_started":1,"wal-append.queue_time_us":715907}
I20260202 15:08:44.828714 14918 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.829231 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.709563 13363 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 1.
W20260202 15:08:44.708746 12726 rpcz_store.cc:269] 0202 15:08:43.382207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.400764 (+ 18557us) service_pool.cc:225] Handling call
0202 15:08:44.708257 (+1307493us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b540033bb54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:44.708339 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1977674624}
I20260202 15:08:44.710949 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 307) took 1235 ms. Trace:
I20260202 15:08:44.711982 12641 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:44.830853 12724 rpcz_store.cc:276] 0202 15:08:43.475399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.453924 (+978525us) service_pool.cc:225] Handling call
0202 15:08:44.710879 (+256955us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.830914 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.833092 12647 raft_consensus.cc:1275] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260202 15:08:44.833632 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1498) took 3787 ms. Trace:
I20260202 15:08:44.833855 12647 rpcz_store.cc:276] 0202 15:08:41.046006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:41.046160 (+ 154us) service_pool.cc:225] Handling call
0202 15:08:41.046397 (+ 237us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:43.491688 (+2445291us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b540030bc20. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4d6df0 00007f24ed560e40 00007f24ed561eb1 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c
0202 15:08:44.833410 (+1341722us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.833594 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4034193664}
I20260202 15:08:44.771528 12722 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:44.834656 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.708925 14889 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260202 15:08:44.835739 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 260609us
I20260202 15:08:44.831764 12648 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:44.711553 12642 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:44.705648 13326 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
W20260202 15:08:44.775740 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 54891.2us
I20260202 15:08:44.655229 14979 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.785755 14950 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.780355 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.782032 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.782445 13358 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:44.783061 12721 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:44.783936 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.798883 14924 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:44.850772 12649 raft_consensus.cc:2468] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:44.703368 14865 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:44.687414 14913 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:44.671453 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.691601 14930 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.800189 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.712625 14936 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:44.771859 14885 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:44.803135 14869 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.791420 14960 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.012s
I20260202 15:08:44.805029 14972 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.791625 14868 consensus_queue.cc:1048] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260202 15:08:44.805887 14793 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:44.806524 14502 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.806900 14927 raft_consensus.cc:3055] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:44.775755 14926 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:44.876029 14926 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:44.877924 14926 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:44.808554 12725 rpcz_store.cc:276] 0202 15:08:43.543207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.473434 (+930227us) service_pool.cc:225] Handling call
0202 15:08:44.701137 (+227703us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:44.878638 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:44.879122 12725 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:44.880178 12641 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:44.880003 14952 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.880980 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.881592 12641 raft_consensus.cc:2393] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:44.780102 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.783145 14722 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.874805 14927 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:44.883481 14927 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.883919 14722 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.812642 14970 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.883984 14511 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.813867 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.871367 14793 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:44.885288 14793 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:44.885450 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.814996 12717 raft_consensus.cc:2368] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
I20260202 15:08:44.886317 12646 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:44.815631 12615 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:44.864475 14885 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:44.816587 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.822830 14854 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.822890 14896 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.826414 14851 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.826637 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:44.896775 12795 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 4
I20260202 15:08:44.306998 13354 rpcz_store.cc:276] 0202 15:08:42.527950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.415598 (+887648us) service_pool.cc:225] Handling call
0202 15:08:43.434331 (+ 18733us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:43.480076 (+ 45745us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:08:43.482110 (+ 2034us) log.cc:844] Serialized 14 byte log entry
0202 15:08:43.904889 (+422779us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:43.904949 (+ 60us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:43.904998 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.905075 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:43.905193 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10192,"prepare.queue_time_us":493,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":25394,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"spinlock_wait_cycles":139809024,"thread_start_us":314745,"threads_started":1,"wal-append.queue_time_us":822183}
I20260202 15:08:44.685838 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:44.836205 14778 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260202 15:08:44.836321 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.896467 14722 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:44.898888 14722 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.899221 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.899554 14851 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.899822 12794 raft_consensus.cc:2393] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:44.900295 12689 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:44.900580 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.900863 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.901077 14722 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.901463 12796 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:44.901695 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.829265 14907 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.902315 12793 raft_consensus.cc:2393] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:44.836930 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:44.845561 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 139972us
I20260202 15:08:44.903048 12726 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:44.903304 14926 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.903891 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:44.903759 14926 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.904875 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:44.905027 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:44.905434 13359 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 5
I20260202 15:08:44.905915 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:44.846627 12690 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:44.905416 14511 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.907070 14926 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:44.907318 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:44.902248 12794 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
W20260202 15:08:44.908295 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:44.908381 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.908648 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:44.909047 13362 raft_consensus.cc:2393] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:44.902211 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.908355 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:44.909920 12796 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:44.910094 12794 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:44.703671 14962 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:44.851280 14847 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:44.851393 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.854023 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.807592 14535 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:44.782078 14902 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:44.912395 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 65880.6us
I20260202 15:08:44.856031 12722 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:44.856818 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.856839 12764 leader_election.cc:400] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4.
I20260202 15:08:44.859149 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:44.835794 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1806) took 2274 ms. Trace:
I20260202 15:08:44.838212 14889 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:44.792255 14975 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.075s
I20260202 15:08:44.685367 14976 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.866801 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:44.795045 14844 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:44.832438 14841 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.852187 14924 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:44.856986 14981 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:44.854457 12719 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:44.873873 14893 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.857147 14918 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:44.858016 14913 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:44.882200 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:44.889178 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 73607.6us
I20260202 15:08:44.810720 14973 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:44.865078 14936 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:44.892748 12725 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:44.893635 12646 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:44.853861 14979 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:44.855334 14802 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260202 15:08:44.897856 12718 raft_consensus.cc:2410] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260202 15:08:44.905490 12726 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:44.847950 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 214429us
I20260202 15:08:44.915179 12764 leader_election.cc:403] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260202 15:08:44.915357 12642 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:44.906531 13354 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:44.971899 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.065850 12795 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:45.136055 12649 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:45.135365 12641 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:44.895764 14980 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:44.947976 12689 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:45.098205 12613 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34792 dropped due to backpressure: service queue is full with 50 items
I20260202 15:08:45.028102 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:44.916550 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.151425 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.151994 12795 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:44.915756 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:44.987582 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:45.055311 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 64414.5us
W20260202 15:08:45.055128 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 57286.2us
I20260202 15:08:44.894912 14952 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.151198 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.116619 13326 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:44.916605 13363 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260202 15:08:45.151808 12648 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:08:45.142807 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 286187us
W20260202 15:08:45.158475 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:45.176604 12764 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:45.176868 12765 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:45.178021 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.178491 12725 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:45.152623 12722 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:45.179023 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.179854 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.180542 12722 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:45.181656 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.183156 12615 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:45.152427 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 204563us
W20260202 15:08:45.024106 14963 log.cc:927] Time spent T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.709s user 0.000s sys 0.002s
I20260202 15:08:45.185822 12689 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:45.187700 12643 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:45.185154 12615 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:45.188892 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 72361.7us
I20260202 15:08:45.190213 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.028230 14927 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:45.192646 12799 raft_consensus.cc:2368] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7.
I20260202 15:08:44.917186 14896 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260202 15:08:45.175547 14846 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.020s sys 0.020s
I20260202 15:08:44.912775 14847 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:45.200080 12689 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:45.194845 12690 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:44.912671 14962 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:45.143061 14924 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:45.142817 14889 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:45.182806 14511 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:45.042742 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:45.198504 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:45.199138 12765 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:45.200407 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.184131 14985 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4.
I20260202 15:08:45.194253 14960 raft_consensus.cc:2804] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:45.135727 14973 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:45.154361 14980 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:45.167831 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1455) took 1719 ms (client timeout 2148 ms). Trace:
I20260202 15:08:44.971423 14915 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.236083 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:45.153947 14930 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:44.909802 14854 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.150170 14981 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.244303 14854 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:45.244997 14981 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:45.246332 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 313) took 1693 ms (client timeout 2175 ms). Trace:
W20260202 15:08:45.246747 12721 rpcz_store.cc:269] 0202 15:08:43.552893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.506111 (+953218us) service_pool.cc:225] Handling call
0202 15:08:45.246167 (+740056us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54002503d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:45.246273 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":362841856}
I20260202 15:08:45.247746 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.249342 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.249816 12650 raft_consensus.cc:1275] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260202 15:08:45.204861 13328 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:45.186004 14502 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:45.141734 14869 raft_consensus.cc:2764] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260202 15:08:44.988781 14969 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:44.964991 14681 consensus_queue.cc:1048] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.231s
I20260202 15:08:45.206624 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:45.229128 13355 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 5
I20260202 15:08:45.257818 12718 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:45.234383 14980 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:45.190130 14851 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:45.073784 14971 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.032280 14793 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.203948 14849 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.221477 14893 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:45.264950 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 65038.1us
I20260202 15:08:45.195793 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:45.200353 14846 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s user 0.021s sys 0.021s
I20260202 15:08:45.200464 14952 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:45.182968 14907 consensus_queue.cc:1059] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260202 15:08:45.277694 14907 raft_consensus.cc:3055] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:45.277993 14907 raft_consensus.cc:740] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:45.278666 14907 consensus_queue.cc:260] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.279524 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.279860 14907 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:45.280635 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.280967 14793 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.230446 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.231993 12765 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:45.234018 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:45.205541 14984 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:45.283917 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.284463 12796 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:45.203701 14847 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:45.200084 14896 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:45.203542 14938 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
W20260202 15:08:45.240857 14957 log.cc:927] Time spent T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 1.335s user 0.003s sys 0.000s
I20260202 15:08:45.240993 14976 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:45.242323 12763 leader_election.cc:336] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Timed out: RequestConsensusVote RPC to 127.11.46.193:40057 timed out after 1.563s (SENT)
W20260202 15:08:45.291790 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 49470.3us
I20260202 15:08:45.281733 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:45.251907 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 47074.2us
W20260202 15:08:45.253633 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 58850.1us
I20260202 15:08:45.256285 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:45.205340 12726 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:45.228250 14960 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:45.260500 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.320786 12794 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:45.321269 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:45.321550 12794 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:45.235673 14258 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:45.239329 14883 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:45.217641 14835 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:45.238653 14902 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:45.239027 14930 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:45.265743 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.205787 14950 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260202 15:08:45.225174 14870 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.254810 14502 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:45.103361 12614 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42782 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1522)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 354)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1527)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1527)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1531)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 351)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1526)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 352)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 356)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1510)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1530)
W20260202 15:08:45.273758 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 62219.4us
I20260202 15:08:45.275182 12689 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:45.276474 12720 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:45.233445 14511 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:45.230443 12719 rpcz_store.cc:269] 0202 15:08:43.431008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.398380 (+967372us) service_pool.cc:225] Handling call
0202 15:08:45.150262 (+751882us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:45.258196 14854 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:45.285324 12718 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:45.286638 13358 raft_consensus.cc:1240] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:45.288812 13361 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
W20260202 15:08:45.286320 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 54404.8us
I20260202 15:08:45.292701 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.196816 14634 consensus_queue.cc:1048] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260202 15:08:45.293500 14979 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:45.294487 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 570) took 1049 ms. Trace:
I20260202 15:08:45.296483 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:45.300276 13359 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
W20260202 15:08:45.301239 12545 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45920 (request call id 308) took 12461 ms (client timeout 15000 ms). Trace:
I20260202 15:08:45.303215 14938 raft_consensus.cc:697] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:45.303650 14787 raft_consensus.cc:1064] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:45.304217 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1800) took 3129 ms. Trace:
I20260202 15:08:45.304518 14849 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:45.288551 14896 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:45.284210 14984 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:45.304957 14847 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:45.306555 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 312) took 1759 ms (client timeout 1890 ms). Trace:
I20260202 15:08:45.271368 14851 raft_consensus.cc:697] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:45.309062 14889 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:45.324133 12763 leader_election.cc:336] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Timed out: RequestConsensusVote RPC to 127.11.46.193:40057 timed out after 1.645s (SENT)
I20260202 15:08:45.320245 13363 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:45.327492 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.319569 13328 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:45.150890 12613 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34792 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1522)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 354)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1527)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1527)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1531)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 351)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1526)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 352)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 356)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1510)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1530)
W20260202 15:08:45.404834 14922 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.097s user 0.009s sys 0.032s
I20260202 15:08:45.458384 14970 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:45.656383 14922 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.351s user 0.010s sys 0.040s
I20260202 15:08:45.656494 14907 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:45.657487 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.657313 14907 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:45.657764 14511 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.325799 14960 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:45.659420 14907 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:45.593677 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.454452 12546 catalog_manager.cc:5654] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:45.675334 12613 service_pool.cc:138] ChangeConfig request on kudu.consensus.ConsensusService from 127.0.0.1:34778 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1527)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1531)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 351)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1526)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 352)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 356)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1536)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1510)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1530)
I20260202 15:08:45.666783 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1454) took 2258 ms. Trace:
I20260202 15:08:45.660812 14935 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:45.660619 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:45.661216 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 341715us
I20260202 15:08:45.683763 14907 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:45.684186 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:45.684301 14979 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:45.684916 13328 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:45.684962 14907 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:45.685747 12615 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:45.661777 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:45.685949 13328 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:45.686456 13361 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 3
I20260202 15:08:45.686887 14950 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:45.687214 14907 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:45.687410 14950 raft_consensus.cc:697] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:45.687595 14907 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:45.687985 14907 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:45.663761 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 388617us
I20260202 15:08:45.686456 14846 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260202 15:08:45.688931 14846 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:45.689354 14846 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:45.658988 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.690595 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.691169 12719 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:45.663102 14992 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:45.692232 14992 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:45.692346 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.449285 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.677824 12718 rpcz_store.cc:276] 0202 15:08:43.408465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.194885 (+1786420us) service_pool.cc:225] Handling call
0202 15:08:45.666726 (+471841us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:45.591360 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.693713 12647 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:45.694043 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.694432 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.694662 14847 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:45.694372 14907 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:45.695179 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:45.695447 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.695747 12718 raft_consensus.cc:2368] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3.
I20260202 15:08:45.695921 12647 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:45.696332 14907 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:45.666107 14869 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:45.696434 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:45.696920 14869 consensus_queue.cc:1243] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260202 15:08:45.364015 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:45.426965 14844 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.361754 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.548525 14936 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.701689 13328 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:45.702344 12724 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:45.702740 13361 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:45.703627 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:45.664424 12545 rpcz_store.cc:269] 0202 15:08:32.839542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:32.849379 (+ 9837us) service_pool.cc:225] Handling call
0202 15:08:45.301177 (+12451798us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:08:40.183178 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:08:40.183972 (+ 794us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:08:40.184019 (+ 47us) write_op.cc:435] Acquired schema lock
0202 15:08:40.184036 (+ 17us) tablet.cc:660] Decoding operations
0202 15:08:40.185017 (+ 981us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:08:40.185116 (+ 99us) write_op.cc:641] Partition lock acquired for write op
0202 15:08:40.185142 (+ 26us) tablet.cc:683] Acquiring locks for 9 operations
0202 15:08:40.186097 (+ 955us) tablet.cc:699] Row locks acquired
0202 15:08:40.186127 (+ 30us) write_op.cc:260] PREPARE: finished
0202 15:08:40.186301 (+ 174us) write_op.cc:270] Start()
0202 15:08:40.186466 (+ 165us) write_op.cc:276] Timestamp: P: 1770044920186225 usec, L: 0
0202 15:08:40.186489 (+ 23us) op_driver.cc:348] REPLICATION: starting
0202 15:08:40.273352 (+ 86863us) log.cc:844] Serialized 3670 byte log entry
0202 15:08:40.608828 (+335476us) op_driver.cc:464] REPLICATION: finished
0202 15:08:40.791632 (+182804us) write_op.cc:301] APPLY: starting
0202 15:08:40.798841 (+ 7209us) tablet.cc:1365] starting BulkCheckPresence
0202 15:08:40.840008 (+ 41167us) tablet.cc:1368] finished BulkCheckPresence
0202 15:08:40.854818 (+ 14810us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:08:40.873612 (+ 18794us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:08:40.973832 (+100220us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0202 15:08:40.979519 (+ 5687us) write_op.cc:312] APPLY: finished
0202 15:08:41.037918 (+ 58399us) log.cc:844] Serialized 71 byte log entry
0202 15:08:41.059177 (+ 21259us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:41.086269 (+ 27092us) write_op.cc:454] Released schema lock
0202 15:08:41.090118 (+ 3849us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":2613176,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":163040,"apply.run_cpu_time_us":14652,"apply.run_wall_time_us":361535,"num_ops":9,"prepare.queue_time_us":174262,"prepare.run_cpu_time_us":4679,"prepare.run_wall_time_us":127003,"replication_time_us":281023,"thread_start_us":152954,"threads_started":2}]]}
I20260202 15:08:45.435635 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.704595 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.704779 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.705107 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:45.705431 12647 raft_consensus.cc:2368] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:45.705623 12649 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:45.325891 14258 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:45.706941 14258 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:45.454689 14924 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.633837 14980 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.708346 14924 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.668596 12723 raft_consensus.cc:2393] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:45.708963 14846 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.709556 14980 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.709702 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:45.710392 12723 raft_consensus.cc:2393] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:45.666837 14930 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:45.389066 14885 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:45.705114 12724 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:45.711148 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.711725 12723 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:45.712239 14885 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:45.712472 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.712677 12649 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:45.390292 14913 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:45.712944 12723 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:45.629738 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.713380 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.713984 12649 raft_consensus.cc:2393] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:45.714133 14913 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:45.714170 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.540180 13354 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:45.469151 12794 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:45.711663 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.715232 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:45.715369 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.715958 12724 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:45.715996 12797 raft_consensus.cc:2393] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:45.716602 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1435) took 3911 ms. Trace:
I20260202 15:08:45.717026 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.593027 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1508) took 1062 ms. Trace:
I20260202 15:08:45.435329 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.445942 14962 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.717743 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1517) took 1119 ms. Trace:
I20260202 15:08:45.717959 12793 raft_consensus.cc:2393] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:45.718138 12643 rpcz_store.cc:276] 0202 15:08:44.598113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.716802 (+1118689us) service_pool.cc:225] Handling call
0202 15:08:45.717698 (+ 896us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:45.716967 12724 rpcz_store.cc:276] 0202 15:08:41.804797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.715588 (+3910791us) service_pool.cc:225] Handling call
0202 15:08:45.715822 (+ 234us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:45.716325 (+ 503us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:45.716552 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:45.718722 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.718961 14962 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:45.717626 12645 rpcz_store.cc:276] 0202 15:08:44.530127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.530260 (+ 133us) service_pool.cc:225] Handling call
0202 15:08:45.592855 (+1062595us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54001dfbd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:45.592966 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":417528576}
I20260202 15:08:45.719750 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.719944 13357 raft_consensus.cc:1240] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260202 15:08:45.625783 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.720319 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.720619 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.720597 12641 raft_consensus.cc:2393] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:45.663340 14502 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:45.668910 12765 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:45.721202 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.721391 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:45.721737 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 52891.4us
I20260202 15:08:45.587994 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.721827 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.672742 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:45.591360 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 560) took 3505 ms. Trace:
I20260202 15:08:45.673288 12688 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:45.658885 12796 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:45.722910 12646 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 2.
I20260202 15:08:45.723704 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.723821 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.724490 12796 raft_consensus.cc:2410] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 4 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260202 15:08:45.725267 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.725769 12796 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:45.722402 12765 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:45.332171 12791 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:45.726964 12641 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 2.
I20260202 15:08:45.727524 14913 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:45.727768 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1480) took 1154 ms. Trace:
I20260202 15:08:45.727845 14913 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:45.728299 14913 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:45.728157 12791 rpcz_store.cc:276] 0202 15:08:44.573202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.654794 (+ 81592us) service_pool.cc:225] Handling call
0202 15:08:45.727607 (+1072813us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54002ebb54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:45.727719 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":919258496}
I20260202 15:08:45.693010 12719 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
W20260202 15:08:45.730314 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 57090.8us
I20260202 15:08:45.730661 12719 raft_consensus.cc:1275] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260202 15:08:45.727871 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.731072 12719 pending_rounds.cc:85] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Aborting all ops after (but not including) 3
I20260202 15:08:45.731362 12719 pending_rounds.cc:107] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260202 15:08:45.731575 12719 raft_consensus.cc:2889] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:45.291963 14762 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:45.731688 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:45.448663 14958 log.cc:927] Time spent T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 1.108s user 0.002s sys 0.000s
I20260202 15:08:45.732169 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1439) took 3549 ms. Trace:
I20260202 15:08:45.663102 14984 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:45.721252 12643 raft_consensus.cc:2376] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5. Re-sending same reply.
I20260202 15:08:45.721884 12645 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:45.733666 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.733865 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:45.734153 12643 raft_consensus.cc:2468] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:45.734397 12645 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:45.734891 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:45.735415 12643 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:45.327793 12690 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:45.739359 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 411600us
I20260202 15:08:45.740373 14913 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.740746 12690 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:45.740958 13328 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:45.383915 14991 consensus_queue.cc:1048] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.311s
I20260202 15:08:45.677210 12650 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:45.661576 14835 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:45.741974 14835 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:45.741863 14896 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.677755 14935 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:45.722987 13356 rpcz_store.cc:276] 0202 15:08:42.085571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.101338 (+1015767us) service_pool.cc:225] Handling call
0202 15:08:43.101761 (+ 423us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:43.281319 (+179558us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:43.281349 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:43.372209 (+ 90860us) log.cc:844] Serialized 26 byte log entry
0202 15:08:43.477736 (+105527us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:43.477793 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:43.477902 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:45.238202 (+1760300us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:45.586783 (+348581us) raft_consensus.cc:1685] finished
0202 15:08:45.586824 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:45.591244 (+ 4420us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":224094,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"spinlock_wait_cycles":3393408,"thread_start_us":224653,"threads_started":2,"wal-append.queue_time_us":507195}
I20260202 15:08:45.710768 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.749925 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:45.677757 12613 service_pool.cc:138] ChangeConfig request on kudu.consensus.ConsensusService from 127.0.0.1:34778 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1527)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1531)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 351)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1526)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 352)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 356)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1536)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1510)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1530)
I20260202 15:08:45.678786 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:45.327426 14883 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:45.682600 14854 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:45.738384 14984 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:45.648047 12764 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:45.663699 14849 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:45.663157 12800 rpcz_store.cc:276] 0202 15:08:42.175023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.504892 (+1329869us) service_pool.cc:225] Handling call
0202 15:08:43.505141 (+ 249us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:44.346800 (+841659us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:45.304015 (+957215us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b54003728d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:45.304167 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1700340992}
I20260202 15:08:45.663893 13360 rpcz_store.cc:276] 0202 15:08:44.244872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.394539 (+149667us) service_pool.cc:225] Handling call
0202 15:08:45.294352 (+899813us) spinlock_profiling.cc:243] Waited 90.2 ms on lock 0x7b540064ffd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:45.294439 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207509632}
I20260202 15:08:45.685405 14960 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.580574 12642 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:45.688236 14950 consensus_queue.cc:237] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.656406 14938 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.692605 14992 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:45.693063 12799 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:45.598975 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.701758 14936 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.668707 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:45.704216 14942 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:45.665956 12614 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42782 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1527)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1531)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 351)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1526)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 352)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 356)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1536)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1510)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1530)
I20260202 15:08:45.717245 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1481) took 1143 ms. Trace:
I20260202 15:08:45.714583 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:45.712906 12798 raft_consensus.cc:2393] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:45.135000 12792 rpcz_store.cc:276] 0202 15:08:42.538109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.341195 (+1803086us) service_pool.cc:225] Handling call
0202 15:08:44.417517 (+ 76322us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:44.466121 (+ 48604us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:44.466175 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:44.466635 (+ 460us) log.cc:844] Serialized 26 byte log entry
0202 15:08:44.574068 (+107433us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:44.598494 (+ 24426us) log.cc:844] Serialized 14 byte log entry
0202 15:08:44.650830 (+ 52336us) spinlock_profiling.cc:243] Waited 4.36 ms on lock 0x7b1800085720. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea4b12d3 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad 0000000000379254 0000000000379099 00007f24ea49469a
0202 15:08:44.653838 (+ 3008us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b1800085720. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea4b1402 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad 0000000000379254 0000000000379099 00007f24ea49469a
0202 15:08:44.778853 (+125015us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:44.781331 (+ 2478us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:44.791652 (+ 10321us) raft_consensus.cc:1685] finished
0202 15:08:44.791689 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:44.812182 (+ 20493us) inbound_call.cc:173] Queueing success response
0202 15:08:45.096717 (+284535us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b4800281dd8. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f2478e21 00007f24f246feb7 00007f24f246fd4a 00007f24f2566f7f 00007f24f2566c47 00007f24f257b26c 00007f24f257b218 00007f24f257b198 00007f24f257b153 00007f24f257a3fc 00007f24f14b55b6
Metrics: {"mutex_wait_us":3778,"prepare.queue_time_us":9572,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":36,"raft.queue_time_us":23417,"raft.run_cpu_time_us":2896,"raft.run_wall_time_us":472250,"spinlock_wait_cycles":66505344,"thread_start_us":97691,"threads_started":1,"wal-append.queue_time_us":144484}
I20260202 15:08:45.722805 14511 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:45.592132 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.724424 12646 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:45.726922 12765 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:45.449129 14865 consensus_meta.cc:287] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.585s user 0.003s sys 0.004s
I20260202 15:08:45.447608 13355 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:45.732115 14762 raft_consensus.cc:697] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:08:45.659317 12717 rpcz_store.cc:269] 0202 15:08:43.547224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.342280 (+795056us) service_pool.cc:225] Handling call
0202 15:08:45.306362 (+964082us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b54003acfd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:45.306495 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":951963520}
I20260202 15:08:45.742312 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:45.742573 14847 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:45.732228 12641 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:45.662444 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 315) took 1983 ms. Trace:
I20260202 15:08:45.387573 14973 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:45.716279 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:45.743783 14869 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:45.744318 14634 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:45.671197 14851 consensus_queue.cc:237] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:45.722445 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:45.678125 12546 catalog_manager.cc:5654] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 4 to 5, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 5 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:45.664584 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 340452us
I20260202 15:08:45.670689 14889 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:45.666261 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:45.747262 12645 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:45.745257 12720 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:45.750257 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:46.214597 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 487705us
I20260202 15:08:46.215556 12765 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:45.895546 12688 leader_election.cc:336] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Timed out: RequestConsensusVote RPC to 127.11.46.193:40057 timed out after 1.643s (SENT)
I20260202 15:08:45.732627 12719 rpcz_store.cc:276] 0202 15:08:42.183062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.730148 (+3547086us) service_pool.cc:225] Handling call
0202 15:08:45.730393 (+ 245us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:45.731974 (+ 1581us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:45.732125 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.200587 14869 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:46.195187 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 547202us
I20260202 15:08:46.218156 14869 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:45.846169 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 105459us
I20260202 15:08:46.218832 12764 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:45.858723 12615 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:46.195933 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1514) took 1309 ms. Trace:
I20260202 15:08:45.796731 12643 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:46.220093 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1518) took 1332 ms. Trace:
I20260202 15:08:46.219920 12646 rpcz_store.cc:276] 0202 15:08:44.886831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.723589 (+836758us) service_pool.cc:225] Handling call
0202 15:08:46.195878 (+472289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.220541 12643 rpcz_store.cc:276] 0202 15:08:44.887807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.734685 (+846878us) service_pool.cc:225] Handling call
0202 15:08:46.220044 (+485359us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.221099 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.196679 14847 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:46.221838 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1535) took 1124 ms. Trace:
I20260202 15:08:46.221319 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.222252 12646 rpcz_store.cc:276] 0202 15:08:45.097756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.220816 (+1123060us) service_pool.cc:225] Handling call
0202 15:08:46.221781 (+ 965us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.222572 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 356) took 1285 ms. Trace:
I20260202 15:08:46.223651 12647 rpcz_store.cc:276] 0202 15:08:44.936694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.750015 (+813321us) service_pool.cc:225] Handling call
0202 15:08:46.222518 (+472503us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.196949 14762 consensus_queue.cc:237] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:46.197961 12648 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1520) took 1309 ms (client timeout 1705 ms). Trace:
I20260202 15:08:46.198221 14960 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:46.198526 14973 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:46.198993 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1508) took 1313 ms. Trace:
I20260202 15:08:46.199981 14936 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:46.199812 14634 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:45.754837 14681 consensus_queue.cc:1048] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.446s
I20260202 15:08:46.242198 12641 rpcz_store.cc:276] 0202 15:08:44.885229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.731505 (+846276us) service_pool.cc:225] Handling call
0202 15:08:46.198943 (+467438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:45.758577 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:46.243127 13361 raft_consensus.cc:2368] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7.
I20260202 15:08:46.243136 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.194941 12650 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:46.251034 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 352) took 1403 ms. Trace:
I20260202 15:08:46.251446 12650 rpcz_store.cc:276] 0202 15:08:44.847175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.742102 (+894927us) service_pool.cc:225] Handling call
0202 15:08:46.250957 (+508855us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.252465 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.253221 12650 raft_consensus.cc:2393] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:46.253614 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1512) took 1367 ms. Trace:
I20260202 15:08:46.198530 12797 rpcz_store.cc:276] 0202 15:08:44.573485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.672678 (+ 99193us) service_pool.cc:225] Handling call
0202 15:08:45.717063 (+1044385us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001f8bd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:45.717193 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2396982656}
I20260202 15:08:46.198662 12721 rpcz_store.cc:276] 0202 15:08:43.679527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.257061 (+1577534us) service_pool.cc:225] Handling call
0202 15:08:45.662394 (+405333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.196009 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:46.256330 13360 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:46.256798 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 580) took 1062 ms. Trace:
I20260202 15:08:46.257138 13360 rpcz_store.cc:276] 0202 15:08:45.194409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.195751 (+1001342us) service_pool.cc:225] Handling call
0202 15:08:46.256743 (+ 60992us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.202857 14854 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:46.212844 14883 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:46.221645 14885 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:46.205626 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 623) took 1300 ms. Trace:
I20260202 15:08:46.227124 14847 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:46.238361 14924 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:46.206935 13362 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:46.208346 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 337) took 1178 ms. Trace:
I20260202 15:08:46.307932 12795 rpcz_store.cc:276] 0202 15:08:45.030243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.743472 (+713229us) service_pool.cc:225] Handling call
0202 15:08:46.206280 (+462808us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.206579 (+ 299us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:46.206609 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:46.206652 (+ 43us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:46.206681 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.206760 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:46.208197 (+ 1437us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b540036db50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:46.208295 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2059,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"spinlock_wait_cycles":1063658880}
I20260202 15:08:45.919976 14835 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:46.198351 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1522) took 1394 ms (client timeout 1611 ms). Trace:
W20260202 15:08:46.238257 12720 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1460) took 1719 ms (client timeout 1966 ms). Trace:
I20260202 15:08:46.210548 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1529) took 1405 ms. Trace:
I20260202 15:08:46.200487 13358 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260202 15:08:46.313629 12649 rpcz_store.cc:276] 0202 15:08:44.805572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.714396 (+908824us) service_pool.cc:225] Handling call
0202 15:08:46.210433 (+496037us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.313957 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 627) took 1408 ms. Trace:
I20260202 15:08:46.254189 14511 raft_consensus.cc:697] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:46.196900 12546 catalog_manager.cc:5654] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:46.316764 12689 leader_election.cc:400] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7.
I20260202 15:08:46.317066 12689 leader_election.cc:403] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260202 15:08:46.217403 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 321856us
I20260202 15:08:45.753540 12726 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:46.224006 12643 raft_consensus.cc:2393] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.318598 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1530) took 1512 ms. Trace:
I20260202 15:08:46.318897 12643 rpcz_store.cc:276] 0202 15:08:44.805833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.221124 (+1415291us) service_pool.cc:225] Handling call
0202 15:08:46.318551 (+ 97427us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.224439 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.319898 12647 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:46.320583 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1529) took 1224 ms. Trace:
I20260202 15:08:46.320963 12647 rpcz_store.cc:276] 0202 15:08:45.096095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.224260 (+1128165us) service_pool.cc:225] Handling call
0202 15:08:46.320519 (+ 96259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.321908 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:46.227917 12648 rpcz_store.cc:269] 0202 15:08:44.888280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.598679 (+710399us) service_pool.cc:225] Handling call
0202 15:08:46.197908 (+599229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.228257 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 574) took 1040 ms. Trace:
I20260202 15:08:46.323414 13363 rpcz_store.cc:276] 0202 15:08:45.187883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.678573 (+490690us) service_pool.cc:225] Handling call
0202 15:08:46.228186 (+549613us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.230942 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1479) took 1658 ms. Trace:
I20260202 15:08:46.324365 12798 rpcz_store.cc:276] 0202 15:08:44.572874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.709488 (+136614us) service_pool.cc:225] Handling call
0202 15:08:46.228067 (+1518579us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b54001f86d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:46.230875 (+ 2808us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1984045312}
I20260202 15:08:46.253937 12650 rpcz_store.cc:276] 0202 15:08:44.886324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.252170 (+1365846us) service_pool.cc:225] Handling call
0202 15:08:46.253563 (+ 1393us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.202119 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:45.849925 14907 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:45.760643 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:45.761153 14846 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:45.754635 14896 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:46.265596 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.261274 14634 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:46.276125 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:46.205411 13357 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 1 FOLLOWER]: Advancing to term 3
I20260202 15:08:46.356796 12646 raft_consensus.cc:2368] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:46.218802 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:46.279101 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.206164 14942 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:46.295032 14870 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:46.297652 14973 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:46.281276 13355 rpcz_store.cc:276] 0202 15:08:44.905010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.042386 (+137376us) service_pool.cc:225] Handling call
0202 15:08:46.205490 (+1163104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.224059 14865 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:46.305438 12613 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42782 dropped due to backpressure: service queue is full with 50 items [suppressed 4 similar messages]
I20260202 15:08:46.221840 13354 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
W20260202 15:08:46.309563 12642 rpcz_store.cc:269] 0202 15:08:44.803671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.846204 (+ 42533us) service_pool.cc:225] Handling call
0202 15:08:46.198299 (+1352095us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:46.199023 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1527) took 1394 ms (client timeout 1795 ms). Trace:
W20260202 15:08:46.391275 12644 rpcz_store.cc:269] 0202 15:08:44.805013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.591852 (+786839us) service_pool.cc:225] Handling call
0202 15:08:46.198967 (+607115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.258625 14936 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:45.758183 14935 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:46.313441 12720 rpcz_store.cc:269] 0202 15:08:44.507944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:44.912890 (+404946us) service_pool.cc:225] Handling call
0202 15:08:46.227432 (+1314542us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.211905 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1521) took 1408 ms. Trace:
I20260202 15:08:46.314646 13328 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:46.404004 12645 rpcz_store.cc:276] 0202 15:08:44.803343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.733673 (+930330us) service_pool.cc:225] Handling call
0202 15:08:46.211844 (+478171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.261386 14913 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:46.315909 12793 raft_consensus.cc:2393] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:46.283061 14915 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260202 15:08:46.404840 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 90420.1us
I20260202 15:08:46.404892 14846 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:46.317834 12546 catalog_manager.cc:5654] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 5 to 6, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 6 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
I20260202 15:08:46.405139 14915 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:46.405964 14915 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:46.406850 13328 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:46.406482 12546 catalog_manager.cc:5654] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 2 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } }
W20260202 15:08:46.408303 13328 leader_election.cc:336] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Timed out: RequestConsensusVote RPC to 127.11.46.194:44367 timed out after 1.821s (SENT)
I20260202 15:08:46.408674 13328 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:46.408514 12546 catalog_manager.cc:5654] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 2, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 2 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:46.369565 12613 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42782 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1536)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 364)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1544)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1545)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 370)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 369)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1547)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1523)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1551)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:58672 (request call id 14)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1510)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1537)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1546)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 365)
I20260202 15:08:46.307018 14924 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:46.410123 14924 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:46.412909 14915 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:46.338577 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:46.414624 14915 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:46.340251 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.415670 12643 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:46.333920 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.417141 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.417748 12650 raft_consensus.cc:2376] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4. Re-sending same reply.
I20260202 15:08:46.318933 14885 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:46.355643 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.357343 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 357) took 1420 ms. Trace:
I20260202 15:08:46.419126 12646 rpcz_store.cc:276] 0202 15:08:44.936983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.223662 (+1286679us) service_pool.cc:225] Handling call
0202 15:08:46.357278 (+133616us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.419348 14846 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:08:46.420233 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:46.367657 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:46.369055 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 324) took 1511 ms. Trace:
I20260202 15:08:46.421104 12646 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:46.421363 12726 rpcz_store.cc:276] 0202 15:08:44.857589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.327276 (+469687us) service_pool.cc:225] Handling call
0202 15:08:46.368828 (+1041552us) spinlock_profiling.cc:243] Waited 89.2 ms on lock 0x7b5400145354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:46.368988 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":205197568}
I20260202 15:08:46.422102 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.422339 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.418565 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.423400 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.367465 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:46.384747 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.425024 12721 raft_consensus.cc:2410] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260202 15:08:46.424443 12546 catalog_manager.cc:5654] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:46.425685 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:46.385197 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 186737us
I20260202 15:08:46.426743 12646 raft_consensus.cc:2376] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4. Re-sending same reply.
I20260202 15:08:46.309181 14869 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:46.426518 12546 catalog_manager.cc:5654] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:46.427917 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.428656 14869 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:46.428699 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1532) took 1622 ms. Trace:
I20260202 15:08:46.429257 12646 rpcz_store.cc:276] 0202 15:08:44.806414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.427620 (+1621206us) service_pool.cc:225] Handling call
0202 15:08:46.428642 (+ 1022us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.314340 13358 rpcz_store.cc:276] 0202 15:08:44.906008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.199665 (+1293657us) service_pool.cc:225] Handling call
0202 15:08:46.200065 (+ 400us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.313731 (+113666us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.313908 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.401520 13362 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:46.334867 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.423943 12726 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:46.430861 14984 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:46.431150 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.431483 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.431983 12726 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:46.432370 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.432638 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.433044 12649 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:46.419133 12648 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:46.434198 14885 raft_consensus.cc:697] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:46.426667 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.434521 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.434926 12721 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:46.435267 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1519) took 1547 ms. Trace:
I20260202 15:08:46.435092 14885 consensus_queue.cc:237] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:46.435638 12648 rpcz_store.cc:276] 0202 15:08:44.888042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.434216 (+1546174us) service_pool.cc:225] Handling call
0202 15:08:46.435216 (+ 1000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.435715 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.433255 12726 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:46.436565 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.437110 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.437887 12726 raft_consensus.cc:2393] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:46.359557 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.438161 12648 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:46.438482 12647 raft_consensus.cc:2393] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.438871 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1511) took 1552 ms. Trace:
I20260202 15:08:46.439198 12647 rpcz_store.cc:276] 0202 15:08:44.886066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.329196 (+1443130us) service_pool.cc:225] Handling call
0202 15:08:46.438814 (+109618us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:46.438649 12763 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:46.439958 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:46.440070 12763 leader_election.cc:336] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Timed out: RequestConsensusVote RPC to 127.11.46.193:40057 timed out after 1.795s (SENT)
I20260202 15:08:46.440868 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.441368 12647 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:46.441502 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:46.442075 12798 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:46.442245 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:46.443243 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:46.443872 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:46.444324 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:46.444521 12796 raft_consensus.cc:1240] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:46.445068 12795 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:46.445533 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:46.445343 14787 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:46.409505 14962 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:46.446486 14865 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:46.446547 14962 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:46.447156 14962 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:46.447410 12725 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:46.447355 14984 consensus_queue.cc:1048] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:46.362988 14942 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:46.443900 12798 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:46.437749 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.423679 12650 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.428380 12546 catalog_manager.cc:5654] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 3 to 5, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 5 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:46.444588 12800 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.449501 14984 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:46.450311 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.450428 14835 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:46.449461 12649 raft_consensus.cc:2393] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:46.450925 15002 consensus_queue.cc:1059] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260202 15:08:46.445302 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:46.452039 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.452232 12763 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:46.453058 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.452275 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:46.361007 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 145427us
I20260202 15:08:46.453964 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.455862 12644 raft_consensus.cc:2393] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.454962 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.456791 12642 raft_consensus.cc:2376] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5. Re-sending same reply.
I20260202 15:08:46.451373 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.457952 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.467103 12650 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 6
I20260202 15:08:46.458494 12765 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:46.401654 14896 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:46.391000 14849 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:46.458029 12649 raft_consensus.cc:2393] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.471266 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1520) took 1668 ms. Trace:
I20260202 15:08:46.471689 12649 rpcz_store.cc:276] 0202 15:08:44.802986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.451177 (+1648191us) service_pool.cc:225] Handling call
0202 15:08:46.471207 (+ 20030us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.460889 12726 raft_consensus.cc:2393] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:46.463101 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1486) took 1271 ms. Trace:
I20260202 15:08:46.463816 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.476675 12642 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:46.465234 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.477900 12644 raft_consensus.cc:2393] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.441789 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1515) took 1554 ms. Trace:
I20260202 15:08:46.467839 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.402537 14936 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:46.403327 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 573) took 1216 ms. Trace:
I20260202 15:08:46.478754 12647 rpcz_store.cc:276] 0202 15:08:44.887081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.440664 (+1553583us) service_pool.cc:225] Handling call
0202 15:08:46.441742 (+ 1078us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.479417 12646 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.470567 12765 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:46.481653 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.482203 12644 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:46.483738 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.485351 12763 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:46.485760 12643 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:46.486649 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.487205 12763 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:46.451264 12546 catalog_manager.cc:5654] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 2 to 5, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 5 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260202 15:08:46.488154 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 617610us
I20260202 15:08:46.513823 12615 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:46.489354 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.514752 12644 raft_consensus.cc:2393] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.479821 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.518004 12643 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.490942 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1440) took 4301 ms. Trace:
I20260202 15:08:46.479894 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 359) took 1420 ms. Trace:
I20260202 15:08:46.527012 12646 rpcz_store.cc:276] 0202 15:08:45.059151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.429884 (+1370733us) service_pool.cc:225] Handling call
0202 15:08:46.479835 (+ 49951us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.363236 14681 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:46.480588 13359 rpcz_store.cc:276] 0202 15:08:45.187204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.665141 (+477937us) service_pool.cc:225] Handling call
0202 15:08:45.760569 (+ 95428us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b5c003f000c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f0ecb6c1 00007f24f106c069 00007f24f25144e6 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:46.400188 (+639619us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54056192}
I20260202 15:08:46.532622 14681 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:46.533155 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:46.503356 14258 consensus_meta.cc:287] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.759s user 0.009s sys 0.000s
I20260202 15:08:46.494446 12642 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:46.535125 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.535827 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1525) took 1441 ms. Trace:
I20260202 15:08:46.536233 12642 rpcz_store.cc:276] 0202 15:08:45.094949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.534919 (+1439970us) service_pool.cc:225] Handling call
0202 15:08:46.535766 (+ 847us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.495226 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:46.537147 13328 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:46.537717 13358 raft_consensus.cc:2393] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:08:46.496544 12690 leader_election.cc:336] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.503s (SENT)
W20260202 15:08:46.499154 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 280349us
I20260202 15:08:46.504853 13326 leader_election.cc:400] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:46.538192 13326 leader_election.cc:403] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260202 15:08:46.505450 12765 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:46.506023 13357 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
W20260202 15:08:46.509236 12725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 328) took 1609 ms (client timeout 1751 ms). Trace:
W20260202 15:08:46.539309 12725 rpcz_store.cc:269] 0202 15:08:44.899490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.178883 (+279393us) service_pool.cc:225] Handling call
0202 15:08:46.509084 (+1330201us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54002ff654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:46.509191 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2915368320}
I20260202 15:08:46.540817 12725 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260202 15:08:46.541486 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1724) took 3784 ms. Trace:
I20260202 15:08:46.541812 12725 rpcz_store.cc:276] 0202 15:08:42.757294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.540184 (+3782890us) service_pool.cc:225] Handling call
0202 15:08:46.540537 (+ 353us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:46.541236 (+ 699us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.541432 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.509720 12724 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:46.510258 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:46.507619 12795 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:46.463129 14973 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:46.371905 14870 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:46.480567 14984 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:46.337906 14907 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:46.531991 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.476415 14849 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:46.548467 12646 raft_consensus.cc:2376] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7. Re-sending same reply.
I20260202 15:08:46.535249 12546 catalog_manager.cc:5654] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 1 to 3, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 3 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:08:46.550251 12546 catalog_manager.cc:5654] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 reported cstate change: term changed from 2 to 5, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 5 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:46.497792 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.552115 12645 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:46.538297 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.545370 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.526882 12718 rpcz_store.cc:276] 0202 15:08:42.189331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.737156 (+3547825us) service_pool.cc:225] Handling call
0202 15:08:45.737460 (+ 304us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:45.737643 (+ 183us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:08:46.213598 (+475955us) log.cc:844] Serialized 14 byte log entry
0202 15:08:46.490528 (+276930us) log.cc:844] Serialized 14 byte log entry
0202 15:08:46.490654 (+ 126us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:46.490689 (+ 35us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:46.490763 (+ 74us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.490812 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:46.490896 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":587420,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":1178,"thread_start_us":751987,"threads_started":2}
I20260202 15:08:46.555944 12643 raft_consensus.cc:2393] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.556646 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.557813 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.536270 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.561239 12643 raft_consensus.cc:2410] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:46.546563 12764 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:46.546837 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.564044 12649 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:46.567903 12645 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:08:46.491544 14915 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:46.531930 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:46.590407 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.570873 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.589821 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.592648 12645 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:46.571350 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.593182 12643 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:46.593477 12644 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:46.593770 12643 raft_consensus.cc:606] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:46.534363 14681 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:46.593215 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1523) took 1699 ms. Trace:
I20260202 15:08:46.595072 12645 rpcz_store.cc:276] 0202 15:08:44.893456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.570628 (+1677172us) service_pool.cc:225] Handling call
0202 15:08:46.593159 (+ 22531us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.597720 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.598506 12646 raft_consensus.cc:2393] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:46.599323 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.599973 12647 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:46.600432 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1515) took 2011 ms. Trace:
I20260202 15:08:46.600735 12647 rpcz_store.cc:276] 0202 15:08:44.589265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.555713 (+1966448us) service_pool.cc:225] Handling call
0202 15:08:46.600382 (+ 44669us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.509150 15009 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:46.604925 12644 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:46.610065 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.545120 14913 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260202 15:08:46.548355 14846 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:46.612576 12641 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:46.611771 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 348) took 1936 ms. Trace:
I20260202 15:08:46.618891 12644 rpcz_store.cc:276] 0202 15:08:44.674984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.594185 (+1919201us) service_pool.cc:225] Handling call
0202 15:08:46.604602 (+ 10417us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.611529 (+ 6927us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.611720 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.615147 12645 raft_consensus.cc:1240] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:46.615242 12718 raft_consensus.cc:1240] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:46.620363 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1727) took 3515 ms. Trace:
I20260202 15:08:46.620705 12718 rpcz_store.cc:276] 0202 15:08:43.105651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.555681 (+3450030us) service_pool.cc:225] Handling call
0202 15:08:46.615099 (+ 59418us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.620111 (+ 5012us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.620308 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.561290 14984 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:46.568939 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:46.449542 14511 consensus_queue.cc:237] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:46.534432 14258 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:46.624729 12688 leader_election.cc:336] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Timed out: RequestConsensusVote RPC to 127.11.46.193:40057 timed out after 1.531s (SENT)
I20260202 15:08:46.476454 12793 rpcz_store.cc:276] 0202 15:08:45.191834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.719507 (+527673us) service_pool.cc:225] Handling call
0202 15:08:46.462836 (+743329us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b54001cc0d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:46.463029 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1230513408}
W20260202 15:08:46.628317 12688 leader_election.cc:336] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057): Timed out: RequestConsensusVote RPC to 127.11.46.193:40057 timed out after 1.669s (SENT)
I20260202 15:08:46.629303 12763 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:46.661054 12646 raft_consensus.cc:1240] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:46.662739 12649 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:46.663437 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1507) took 1795 ms. Trace:
I20260202 15:08:46.663580 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1530) took 1567 ms. Trace:
I20260202 15:08:46.663769 12646 rpcz_store.cc:276] 0202 15:08:44.868014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.599030 (+1731016us) service_pool.cc:225] Handling call
0202 15:08:46.660914 (+ 61884us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.663221 (+ 2307us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.663390 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.663939 12649 rpcz_store.cc:276] 0202 15:08:45.096403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.615127 (+1518724us) service_pool.cc:225] Handling call
0202 15:08:46.662556 (+ 47429us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.663386 (+ 830us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.663537 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.611665 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:46.561669 14973 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:46.669443 12763 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:46.621906 14846 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:46.671322 12688 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:46.672907 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1449) took 3388 ms. Trace:
I20260202 15:08:46.677489 12718 rpcz_store.cc:276] 0202 15:08:43.284424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.623668 (+3339244us) service_pool.cc:225] Handling call
0202 15:08:46.672362 (+ 48694us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.672545 (+ 183us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:46.672578 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:46.672630 (+ 52us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:46.672674 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.672737 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:46.672849 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:46.678229 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 131698us
I20260202 15:08:46.680217 12764 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:46.672061 14885 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:46.682401 12688 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:46.683434 14991 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:46.683845 14991 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:46.684250 14991 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:46.691002 12688 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:46.625953 14984 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:46.669445 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 352734us
I20260202 15:08:46.624055 14915 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:46.693830 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.693885 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.695451 12689 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:46.698033 12646 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:46.696264 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 182489us
I20260202 15:08:46.626168 15009 raft_consensus.cc:697] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:46.698922 12615 leader_election.cc:400] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3.
I20260202 15:08:46.699361 12615 leader_election.cc:403] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260202 15:08:46.693437 12688 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:46.720847 12642 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
W20260202 15:08:46.721325 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 182376us
I20260202 15:08:46.719605 12688 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:46.697245 12717 raft_consensus.cc:2368] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7.
I20260202 15:08:46.723884 12718 raft_consensus.cc:1240] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:46.729461 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 306) took 3254 ms. Trace:
I20260202 15:08:46.729780 12718 rpcz_store.cc:276] 0202 15:08:43.475110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.678472 (+3203362us) service_pool.cc:225] Handling call
0202 15:08:46.723665 (+ 45193us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.729213 (+ 5548us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.729407 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:46.672372 15008 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:46.693255 14846 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:46.731263 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 226556us
I20260202 15:08:46.697358 14915 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:46.697346 14681 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:46.731964 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.736523 13326 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:46.703009 15002 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:46.731971 12725 raft_consensus.cc:2955] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:46.747526 13326 leader_election.cc:400] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:46.707125 15023 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:46.725867 14869 raft_consensus.cc:2804] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election won for term 3
W20260202 15:08:46.731455 14992 consensus_meta.cc:287] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.530s user 0.000s sys 0.009s
I20260202 15:08:46.738848 12719 raft_consensus.cc:2393] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:46.731822 14938 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260202 15:08:46.732856 14851 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:46.738468 13356 raft_consensus.cc:2410] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260202 15:08:46.706935 15003 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:46.766047 12646 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:46.765377 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:46.752874 13326 leader_election.cc:403] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260202 15:08:46.772305 12649 raft_consensus.cc:2368] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7.
I20260202 15:08:46.752748 14681 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:46.773221 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.708829 14950 consensus_queue.cc:1059] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260202 15:08:46.750952 14915 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:46.767112 12531 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 256125us
I20260202 15:08:46.751075 15002 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:46.756950 14762 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:08:46.793145 12649 raft_consensus.cc:2410] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260202 15:08:46.821234 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 283314us
I20260202 15:08:46.827322 12649 pending_rounds.cc:85] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 5
I20260202 15:08:46.827682 12649 pending_rounds.cc:107] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260202 15:08:46.828158 12649 raft_consensus.cc:2889] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260202 15:08:46.826298 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 320926us
I20260202 15:08:46.829456 12765 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:46.741806 14924 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:46.848023 13328 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:46.838264 12765 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:46.850353 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 151578us
I20260202 15:08:46.851693 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 630) took 1944 ms. Trace:
W20260202 15:08:46.847520 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 127966us
W20260202 15:08:46.852279 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:46.852735 12688 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:46.853663 15003 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:46.775790 14681 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:46.865350 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 185193us
I20260202 15:08:46.866529 12764 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:46.773734 14869 raft_consensus.cc:697] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:46.792807 12720 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:46.792145 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1510) took 1906 ms. Trace:
I20260202 15:08:46.793402 15002 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:46.765996 14992 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:46.764875 14854 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:46.884775 12648 raft_consensus.cc:2468] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:46.852020 13361 rpcz_store.cc:276] 0202 15:08:44.910759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.243699 (+1332940us) service_pool.cc:225] Handling call
0202 15:08:46.243989 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.244201 (+ 212us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:46.361543 (+117342us) log.cc:844] Serialized 14 byte log entry
0202 15:08:46.851406 (+489863us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:46.851455 (+ 49us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:46.851493 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.851550 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:46.851646 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":158333,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"thread_start_us":598840,"threads_started":3,"wal-append.queue_time_us":408162}
I20260202 15:08:46.886227 12647 rpcz_store.cc:276] 0202 15:08:44.885799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.601299 (+1715500us) service_pool.cc:225] Handling call
0202 15:08:46.624793 (+ 23494us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.791851 (+167058us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:46.791878 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:46.791914 (+ 36us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:46.791940 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.791996 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:46.792096 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":239125,"thread_start_us":151853,"threads_started":1}
I20260202 15:08:46.763360 15023 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:46.904067 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1456) took 3411 ms. Trace:
I20260202 15:08:46.904445 12718 rpcz_store.cc:276] 0202 15:08:43.492717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.730494 (+3237777us) service_pool.cc:225] Handling call
0202 15:08:46.753969 (+ 23475us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.754147 (+ 178us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:08:46.770265 (+ 16118us) log.cc:844] Serialized 14 byte log entry
0202 15:08:46.903718 (+133453us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:46.903765 (+ 47us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:46.903817 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.903888 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:46.904015 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16974,"prepare.run_cpu_time_us":205,"prepare.run_wall_time_us":3053,"thread_start_us":132869,"threads_started":1}
I20260202 15:08:46.888288 12688 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:46.890744 12644 raft_consensus.cc:1240] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260202 15:08:46.888105 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:46.919009 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:46.868523 14907 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:46.868307 14870 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:46.925904 12723 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:46.944160 12718 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:46.948722 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 314) took 3269 ms. Trace:
I20260202 15:08:46.949054 12718 rpcz_store.cc:276] 0202 15:08:43.679216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.905899 (+3226683us) service_pool.cc:225] Handling call
0202 15:08:46.941697 (+ 35798us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.948451 (+ 6754us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.948661 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:46.950028 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 280630us
I20260202 15:08:46.943704 15030 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:46.947882 14973 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:46.950644 15030 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:46.951061 14973 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 2. Current term is 3.
I20260202 15:08:46.951954 14973 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:46.951119 15030 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:46.965869 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 127654us
I20260202 15:08:46.866271 14991 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:46.992051 12650 consensus_meta.cc:287] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.519s user 0.009s sys 0.000s
I20260202 15:08:46.994639 12650 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:46.995543 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.995710 12763 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:46.996078 12650 raft_consensus.cc:2376] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6. Re-sending same reply.
I20260202 15:08:46.997054 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.998126 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:46.998623 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:46.942468 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 309) took 3445 ms. Trace:
I20260202 15:08:46.889477 15027 raft_consensus.cc:2749] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
W20260202 15:08:46.996194 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 148302us
I20260202 15:08:46.946596 14870 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:46.965339 14851 raft_consensus.cc:697] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:47.004340 14885 raft_consensus.cc:697] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:08:46.994863 14634 consensus_meta.cc:287] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.632s user 0.008s sys 0.001s
I20260202 15:08:46.964569 14889 raft_consensus.cc:3055] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:46.964278 15003 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:47.006958 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:46.850301 15009 consensus_queue.cc:237] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:47.090958 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 343565us
I20260202 15:08:47.096853 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1447) took 4365 ms. Trace:
I20260202 15:08:46.963534 15029 raft_consensus.cc:2764] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260202 15:08:47.112618 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:58672
W20260202 15:08:47.091470 15018 log.cc:927] Time spent T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.298s user 0.000s sys 0.000s
I20260202 15:08:47.105453 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1719) took 5298 ms. Trace:
W20260202 15:08:47.096100 14502 consensus_meta.cc:287] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.363s user 0.005s sys 0.005s
I20260202 15:08:47.110342 13326 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:47.110534 12726 rpcz_store.cc:276] 0202 15:08:42.731506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.515594 (+3784088us) service_pool.cc:225] Handling call
0202 15:08:47.096129 (+580535us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b58001d7098. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:47.096348 (+ 219us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:47.096525 (+ 177us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:47.096549 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:47.096594 (+ 45us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:47.096628 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.096685 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.096782 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47820160}
I20260202 15:08:47.039903 14258 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:47.005250 15030 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:47.302812 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:47.315258 15057 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election won for term 6
W20260202 15:08:47.315619 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:47.315752 15057 raft_consensus.cc:697] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:08:47.316407 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 206119us
I20260202 15:08:47.316674 15057 consensus_queue.cc:237] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:47.318723 15055 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:47.319170 15055 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:47.319466 15055 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:47.319816 15035 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:47.320261 15035 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:47.320561 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 432275us
I20260202 15:08:47.321449 12650 raft_consensus.cc:606] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:47.322139 15035 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:47.323228 15001 consensus_queue.cc:1059] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:47.324222 14870 raft_consensus.cc:3055] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:47.324532 14870 raft_consensus.cc:740] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:47.325353 15047 log.cc:927] Time spent T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260202 15:08:47.325448 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1546) took 1123 ms. Trace:
I20260202 15:08:47.325258 14870 consensus_queue.cc:260] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:47.325919 12644 rpcz_store.cc:276] 0202 15:08:46.202074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.619636 (+417562us) service_pool.cc:225] Handling call
0202 15:08:46.890483 (+270847us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:46.921641 (+ 31158us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.325250 (+403609us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b540020dd50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:47.325377 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":616668416}
I20260202 15:08:47.326579 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1537) took 1126 ms. Trace:
W20260202 15:08:47.327931 12615 leader_election.cc:336] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.629s (SENT)
I20260202 15:08:47.327064 12643 rpcz_store.cc:276] 0202 15:08:46.200107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.619021 (+418914us) service_pool.cc:225] Handling call
0202 15:08:46.623332 (+ 4311us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:46.624103 (+ 771us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:46.624137 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:46.662360 (+ 38223us) log.cc:844] Serialized 26 byte log entry
0202 15:08:46.835459 (+173099us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:47.244016 (+408557us) log.cc:844] Serialized 14 byte log entry
0202 15:08:47.244217 (+ 201us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.244299 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:47.326374 (+ 82075us) raft_consensus.cc:1685] finished
0202 15:08:47.326393 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.326520 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":199295,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":72133,"thread_start_us":503477,"threads_started":2,"wal-append.queue_time_us":647932}
I20260202 15:08:47.328092 12724 rpcz_store.cc:276] 0202 15:08:41.806807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.719263 (+3912456us) service_pool.cc:225] Handling call
0202 15:08:46.509577 (+790314us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.544774 (+ 35197us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.105217 (+560443us) spinlock_profiling.cc:243] Waited 790 ms on lock 0x7b54002ff650. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:47.105387 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1816984832}
I20260202 15:08:47.328522 12615 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:47.108114 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 370896us
W20260202 15:08:47.153283 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 157542us
W20260202 15:08:47.102295 15016 log.cc:927] Time spent T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.178s user 0.000s sys 0.000s
W20260202 15:08:47.330394 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 463920us
W20260202 15:08:47.253175 13328 leader_election.cc:336] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Timed out: RequestConsensusVote RPC to 127.11.46.194:44367 timed out after 2.116s (SENT)
I20260202 15:08:47.102362 14889 raft_consensus.cc:740] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:47.323580 15039 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:47.316598 15040 log.cc:927] Time spent T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.185s user 0.000s sys 0.000s
W20260202 15:08:47.263509 15013 log.cc:927] Time spent T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.398s user 0.001s sys 0.001s
I20260202 15:08:47.344746 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1461) took 2824 ms. Trace:
I20260202 15:08:47.340333 12764 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:47.340771 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 87647.1us
W20260202 15:08:47.345093 12531 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 78ef01280730461580740297403fe090 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260202 15:08:47.369410 12688 leader_election.cc:400] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7.
I20260202 15:08:47.207522 15043 raft_consensus.cc:2749] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:47.340466 12614 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:47.380431 12764 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:47.380985 12688 leader_election.cc:403] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260202 15:08:47.338991 15035 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:47.389892 15035 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:47.216662 14924 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:47.381590 15039 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.219552 14851 consensus_queue.cc:237] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.005352 12720 rpcz_store.cc:276] 0202 15:08:43.481413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.698769 (+3217356us) service_pool.cc:225] Handling call
0202 15:08:46.757186 (+ 58417us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.885698 (+128512us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.927241 (+ 41543us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:47.381618 12724 rpcz_store.cc:276] 0202 15:08:44.520525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.343582 (+2823057us) service_pool.cc:225] Handling call
0202 15:08:47.344360 (+ 778us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:47.344498 (+ 138us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:47.344517 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:47.344548 (+ 31us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:47.344570 (+ 22us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.344611 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.344705 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:47.232693 13354 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.016s sys 0.069s
W20260202 15:08:47.154426 14930 consensus_meta.cc:287] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.927s user 0.009s sys 0.001s
W20260202 15:08:47.152091 15006 log.cc:927] Time spent T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.351s user 0.000s sys 0.001s
W20260202 15:08:47.327015 15022 log.cc:927] Time spent T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.205s user 0.001s sys 0.000s
I20260202 15:08:47.104832 14960 raft_consensus.cc:2764] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260202 15:08:47.390326 15035 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:47.390435 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 50108.1us
I20260202 15:08:47.342413 14502 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.262956 15034 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.232s
I20260202 15:08:47.243578 14950 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:47.104209 14634 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:47.346551 15012 log.cc:927] Time spent T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.449s user 0.001s sys 0.000s
I20260202 15:08:47.095803 14992 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:47.232173 15038 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:47.408154 12764 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:47.480437 12726 raft_consensus.cc:1240] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
W20260202 15:08:47.480599 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 72770.6us
I20260202 15:08:47.481705 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.481986 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.299116 14991 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:47.080943 14869 consensus_queue.cc:237] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.294574 15056 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
W20260202 15:08:47.391378 14762 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.006s sys 0.010s
I20260202 15:08:47.407033 15014 consensus_queue.cc:1059] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260202 15:08:47.479801 15038 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:47.152204 15045 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:47.391474 15050 raft_consensus.cc:2764] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
W20260202 15:08:47.391601 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 387747us
I20260202 15:08:47.415602 12644 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
W20260202 15:08:47.450553 13327 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.003s sys 0.017s
W20260202 15:08:47.499030 13327 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.705s user 0.003s sys 0.017s
I20260202 15:08:47.489317 15058 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:47.409122 14258 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:47.406810 15003 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:47.456534 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.415093 15054 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:47.461925 12614 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:47.504416 12644 raft_consensus.cc:606] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:47.391942 15063 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:47.464148 12763 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:47.435592 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 336) took 2386 ms. Trace:
I20260202 15:08:47.454785 14930 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.419435 15043 raft_consensus.cc:2749] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:47.318403 14681 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:47.446913 13354 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.908s user 0.036s sys 0.116s
I20260202 15:08:47.452628 14889 consensus_queue.cc:260] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.517104 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 637) took 1035 ms. Trace:
I20260202 15:08:47.517546 13354 rpcz_store.cc:276] 0202 15:08:46.481263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.533629 (+ 52366us) service_pool.cc:225] Handling call
0202 15:08:46.533874 (+ 245us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.534071 (+ 197us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:46.534326 (+ 255us) log.cc:844] Serialized 14 byte log entry
0202 15:08:47.514962 (+980636us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:47.516765 (+ 1803us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:47.516819 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.516898 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.517046 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11755,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":979162,"threads_started":1}
I20260202 15:08:47.508689 12791 rpcz_store.cc:276] 0202 15:08:45.029990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.729105 (+699115us) service_pool.cc:225] Handling call
0202 15:08:45.729380 (+ 275us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:45.729729 (+ 349us) raft_consensus.cc:1534] Early marking committed up to index 7
0202 15:08:46.192176 (+462447us) log.cc:844] Serialized 14 byte log entry
0202 15:08:46.726517 (+534341us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:46.793258 (+ 66741us) log.cc:844] Serialized 26 byte log entry
0202 15:08:46.863126 (+ 69868us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:46.871901 (+ 8775us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.888692 (+ 16791us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:47.390719 (+502027us) raft_consensus.cc:1685] finished
0202 15:08:47.390754 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.416463 (+ 25709us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3737,"prepare.queue_time_us":632933,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":471097,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":12393,"thread_start_us":557137,"threads_started":2,"wal-append.queue_time_us":1095907}
I20260202 15:08:47.520617 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.521638 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.522287 12791 raft_consensus.cc:2393] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:08:47.523654 14984 consensus_meta.cc:287] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.803s user 0.009s sys 0.000s
I20260202 15:08:47.445683 14924 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:47.471134 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:47.473821 14962 consensus_meta.cc:287] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.026s user 0.003s sys 0.005s
I20260202 15:08:47.294641 14885 consensus_queue.cc:237] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.443428 14935 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:47.477490 12799 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260202 15:08:47.479022 12720 raft_consensus.cc:1240] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:47.482656 12798 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:47.709398 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1736) took 1893 ms. Trace:
I20260202 15:08:47.339366 14854 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:47.451537 15023 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:47.473821 14896 consensus_meta.cc:287] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.993s user 0.005s sys 0.002s
I20260202 15:08:47.711563 14896 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7.
I20260202 15:08:47.711891 12798 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:47.712778 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.713099 12721 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:08:47.712476 12720 rpcz_store.cc:276] 0202 15:08:45.815645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.460969 (+1645324us) service_pool.cc:225] Handling call
0202 15:08:47.478819 (+ 17850us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:47.709170 (+230351us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.709347 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:47.493777 12794 raft_consensus.cc:2368] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:47.094519 14915 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.501989 12795 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:47.714881 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 333) took 1475 ms. Trace:
I20260202 15:08:47.715234 12721 rpcz_store.cc:276] 0202 15:08:46.239465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.435521 (+196056us) service_pool.cc:225] Handling call
0202 15:08:47.714677 (+1279156us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54002503d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:47.714826 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2935998336}
I20260202 15:08:47.715507 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.481400 14960 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:47.505764 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 43928us
I20260202 15:08:47.716106 14915 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:47.716190 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.716869 12721 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:47.717042 15024 consensus_queue.cc:1059] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:47.717278 12614 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:47.717916 15024 raft_consensus.cc:3055] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:47.718307 15024 raft_consensus.cc:740] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:47.392379 12688 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:47.719050 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 326748us
I20260202 15:08:47.719137 14896 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:08:47.719470 15038 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:47.719149 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1737) took 1903 ms (client timeout 1689 ms). Trace:
W20260202 15:08:47.720093 12719 rpcz_store.cc:269] 0202 15:08:45.815991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.218358 (+402367us) service_pool.cc:225] Handling call
0202 15:08:47.718958 (+1500600us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b5400157554. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:47.719073 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":769390976}
I20260202 15:08:47.720980 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.721149 14870 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:47.721633 12719 raft_consensus.cc:2393] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:08:47.721992 14846 consensus_meta.cc:287] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.965s user 0.006s sys 0.001s
I20260202 15:08:47.504804 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 340) took 1063 ms. Trace:
I20260202 15:08:47.509822 14634 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:47.723235 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1438) took 5570 ms. Trace:
I20260202 15:08:47.484809 14950 raft_consensus.cc:697] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:47.723804 12723 rpcz_store.cc:276] 0202 15:08:42.152795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.719803 (+3567008us) service_pool.cc:225] Handling call
0202 15:08:45.720036 (+ 233us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.942573 (+1222537us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.723028 (+780455us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54003007d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:47.723180 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2772762368}
I20260202 15:08:47.724484 14950 consensus_queue.cc:237] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.724959 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.723281 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.725841 12723 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:47.492358 15039 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:47.524091 14930 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:47.524600 12722 raft_consensus.cc:2368] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 2. Current term is 3.
W20260202 15:08:47.525514 14936 consensus_meta.cc:287] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.045s user 0.010s sys 0.000s
W20260202 15:08:47.526315 14849 consensus_meta.cc:287] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.935s user 0.005s sys 0.002s
I20260202 15:08:47.527268 15030 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:47.551885 14889 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:47.527659 15043 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:47.510773 15050 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:47.510372 15054 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.530486 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1725) took 4694 ms. Trace:
W20260202 15:08:47.740517 14942 consensus_meta.cc:287] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.292s user 0.004s sys 0.005s
I20260202 15:08:47.531800 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 317) took 2800 ms. Trace:
I20260202 15:08:47.497848 15056 raft_consensus.cc:697] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:08:47.533391 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 69637.5us
I20260202 15:08:47.508975 12644 raft_consensus.cc:614] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260202 15:08:47.464710 13360 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.024s sys 0.034s
W20260202 15:08:47.769757 13360 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.885s user 0.024s sys 0.034s
I20260202 15:08:47.734001 14258 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7.
W20260202 15:08:47.589761 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:47.695614 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 151989us
I20260202 15:08:47.718116 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.697062 15058 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:47.699523 15027 raft_consensus.cc:3055] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:47.700387 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 349) took 2922 ms. Trace:
I20260202 15:08:47.694831 15001 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:47.702929 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:47.703059 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1518) took 3047 ms (client timeout 2487 ms). Trace:
I20260202 15:08:47.703030 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.705523 12793 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:47.483069 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.798467 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:47.713723 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.705224 15035 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.491559 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:47.718410 15010 log.cc:927] Time spent T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.988s user 0.009s sys 0.042s
W20260202 15:08:47.492657 14762 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s user 0.006s sys 0.011s
I20260202 15:08:47.723477 14502 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:47.715504 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 341) took 1273 ms. Trace:
I20260202 15:08:47.808481 12794 rpcz_store.cc:276] 0202 15:08:46.441914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.442019 (+ 105us) service_pool.cc:225] Handling call
0202 15:08:47.715356 (+1273337us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54002d3f54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:47.715458 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2416806144}
I20260202 15:08:47.808827 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.727747 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.809401 12796 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:47.809628 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.809913 12650 raft_consensus.cc:2393] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:47.810355 12794 raft_consensus.cc:2393] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:08:47.542915 15021 log.cc:927] Time spent T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.691s user 0.000s sys 0.004s
I20260202 15:08:47.736826 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1549) took 1105 ms. Trace:
I20260202 15:08:47.731622 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:47.811923 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.811493 12648 rpcz_store.cc:276] 0202 15:08:46.631086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.891003 (+259917us) service_pool.cc:225] Handling call
0202 15:08:46.891301 (+ 298us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:46.923779 (+ 32478us) raft_consensus.cc:1534] Early marking committed up to index 2
0202 15:08:47.109805 (+186026us) log.cc:844] Serialized 14 byte log entry
0202 15:08:47.627969 (+518164us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:47.735397 (+107428us) log.cc:844] Serialized 26 byte log entry
0202 15:08:47.735517 (+ 120us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:47.735663 (+ 146us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.735721 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:47.736644 (+ 923us) raft_consensus.cc:1685] finished
0202 15:08:47.736662 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.736771 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85322,"prepare.queue_time_us":547238,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":282619,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":19800,"thread_start_us":536316,"threads_started":2,"wal-append.queue_time_us":551063}
I20260202 15:08:47.807178 13328 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
W20260202 15:08:47.743193 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 47964.4us
I20260202 15:08:47.813438 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:47.814152 13326 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.46.193:40057 timed out after 2.456s (SENT) [suppressed 86 similar messages]
W20260202 15:08:47.722553 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:47.719591 14896 raft_consensus.cc:697] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:47.508420 15003 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:47.722901 12797 rpcz_store.cc:276] 0202 15:08:46.441684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.442826 (+ 1142us) service_pool.cc:225] Handling call
0202 15:08:46.443175 (+ 349us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:46.443507 (+ 332us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:46.443544 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:46.611112 (+167568us) log.cc:844] Serialized 26 byte log entry
0202 15:08:46.866626 (+255514us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:46.878597 (+ 11971us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.888724 (+ 10127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:47.497824 (+609100us) raft_consensus.cc:1685] finished
0202 15:08:47.497855 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.504752 (+ 6897us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1353,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"spinlock_wait_cycles":26752,"thread_start_us":136739,"threads_started":1,"wal-append.queue_time_us":331335}
I20260202 15:08:47.722360 14846 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.754263 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1731) took 3365 ms. Trace:
I20260202 15:08:47.757359 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:47.738687 15002 consensus_meta.cc:287] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.830s user 0.005s sys 0.000s
W20260202 15:08:47.739635 14847 consensus_meta.cc:287] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.421s user 0.007s sys 0.003s
I20260202 15:08:47.758312 12725 rpcz_store.cc:276] 0202 15:08:42.835914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.542552 (+3706638us) service_pool.cc:225] Handling call
0202 15:08:46.545280 (+ 2728us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:46.545722 (+ 442us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:46.545751 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:46.586446 (+ 40695us) log.cc:844] Serialized 590 byte log entry
0202 15:08:46.726770 (+140324us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:46.776805 (+ 50035us) log.cc:844] Serialized 14 byte log entry
0202 15:08:46.824895 (+ 48090us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:46.824962 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:47.530293 (+705331us) raft_consensus.cc:1685] finished
0202 15:08:47.530311 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.530435 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32520,"raft.queue_time_us":41943,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":55105,"thread_start_us":65142,"threads_started":1,"wal-append.queue_time_us":863447}
I20260202 15:08:47.531456 12717 raft_consensus.cc:1240] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:47.695695 14681 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:47.760980 12795 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:47.761786 12726 rpcz_store.cc:276] 0202 15:08:44.731473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.329783 (+2598310us) service_pool.cc:225] Handling call
0202 15:08:47.330014 (+ 231us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:47.480815 (+150801us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.531567 (+ 50752us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b5400146754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:47.531740 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":345594112}
W20260202 15:08:47.742359 14511 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.015s sys 0.018s
I20260202 15:08:47.770545 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 581) took 2572 ms. Trace:
I20260202 15:08:47.535634 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1546) took 1070 ms. Trace:
I20260202 15:08:47.832196 12649 rpcz_store.cc:276] 0202 15:08:46.465076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.826819 (+361743us) service_pool.cc:225] Handling call
0202 15:08:46.827152 (+ 333us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:47.110300 (+283148us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:47.110331 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:47.111378 (+ 1047us) log.cc:844] Serialized 52 byte log entry
0202 15:08:47.381214 (+269836us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:47.441886 (+ 60672us) log.cc:844] Serialized 14 byte log entry
0202 15:08:47.443279 (+ 1393us) log.cc:844] Serialized 14 byte log entry
0202 15:08:47.453524 (+ 10245us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.453727 (+ 203us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:47.535413 (+ 81686us) raft_consensus.cc:1685] finished
0202 15:08:47.535436 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.535573 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25779,"prepare.queue_time_us":119791,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":56,"raft.queue_time_us":177459,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":54371,"thread_start_us":597079,"threads_started":3,"wal-append.queue_time_us":341726}
I20260202 15:08:47.696417 15062 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:47.809124 15067 consensus_queue.cc:1059] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:47.831228 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:47.838071 12726 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:47.838915 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.811425 14762 raft_consensus.cc:697] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:47.807348 12791 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:47.777143 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.742833 14924 raft_consensus.cc:697] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:47.737246 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.747103 15038 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:47.783926 12690 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:47.771502 14950 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:47.787297 12645 rpcz_store.cc:276] 0202 15:08:44.778120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.595864 (+1817744us) service_pool.cc:225] Handling call
0202 15:08:46.596141 (+ 277us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.621568 (+ 25427us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.700108 (+1078540us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b54000799d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:47.700319 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43392256}
I20260202 15:08:47.854462 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.737571 15055 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.791424 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:47.795017 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:47.791460 12765 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:47.797216 12641 rpcz_store.cc:269] 0202 15:08:44.655712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.242909 (+1587197us) service_pool.cc:225] Handling call
0202 15:08:47.702885 (+1459976us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b540008a7d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:47.702993 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":847817088}
I20260202 15:08:47.798988 12800 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:47.799475 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.626784 14984 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:08:47.860025 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 1.15915e+06us
I20260202 15:08:47.860330 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.799858 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:47.861828 13361 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:47.716266 14960 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.695489 14962 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.870260 13328 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:47.872153 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.751173 15023 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260202 15:08:47.818228 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.879956 12643 raft_consensus.cc:2376] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7. Re-sending same reply.
I20260202 15:08:47.834355 15026 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:47.787102 15027 raft_consensus.cc:740] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:47.758366 14936 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.817327 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.898406 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.758303 14849 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.822551 12718 rpcz_store.cc:276] 0202 15:08:44.388853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.967894 (+2579041us) service_pool.cc:225] Handling call
0202 15:08:47.031205 (+ 63311us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:47.031578 (+ 373us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:47.031615 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:47.711599 (+679984us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b2c00010828. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ebe5d9b1 00007f24ebe6cfe2 00007f24ed59a851 00007f24ed55f0ba 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22
0202 15:08:47.711938 (+ 339us) log.cc:844] Serialized 26 byte log entry
0202 15:08:47.752388 (+ 40450us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:47.752436 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.752516 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:47.753946 (+ 1430us) raft_consensus.cc:1685] finished
0202 15:08:47.753970 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.754199 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1227,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":36773,"spinlock_wait_cycles":30157184,"thread_start_us":40170,"threads_started":1,"wal-append.queue_time_us":40469}
I20260202 15:08:47.824528 13326 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:47.794955 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:47.719506 15024 consensus_queue.cc:260] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.772533 14942 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.829003 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.822216 12720 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:47.827668 12688 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:47.908972 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 81366.8us
I20260202 15:08:47.909333 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 310) took 4396 ms. Trace:
I20260202 15:08:47.909636 12717 rpcz_store.cc:276] 0202 15:08:43.513220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.751523 (+3238303us) service_pool.cc:225] Handling call
0202 15:08:46.764997 (+ 13474us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:47.828949 (+1063952us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.909149 (+ 80200us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b54002ff154. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:47.909283 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1762302720}
I20260202 15:08:47.830196 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.910837 12648 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:47.833731 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.911408 12649 raft_consensus.cc:2376] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3. Re-sending same reply.
I20260202 15:08:47.912276 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.829770 15002 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:47.832069 13360 rpcz_store.cc:276] 0202 15:08:45.197922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.257734 (+1059812us) service_pool.cc:225] Handling call
0202 15:08:46.258044 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:46.258248 (+ 204us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:46.562688 (+304440us) log.cc:844] Serialized 14 byte log entry
0202 15:08:47.770145 (+1207457us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:47.770212 (+ 67us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:47.770273 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.770353 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.770484 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15397,"prepare.queue_time_us":175383,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":449758,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"thread_start_us":1473276,"threads_started":3,"wal-append.queue_time_us":441725}
I20260202 15:08:47.913686 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:47.766083 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.784071 15043 consensus_queue.cc:1059] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
W20260202 15:08:47.830006 15008 consensus_meta.cc:287] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.079s user 0.010s sys 0.000s
I20260202 15:08:47.800587 15030 consensus_queue.cc:1059] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.031s
I20260202 15:08:47.850965 12644 raft_consensus.cc:2393] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:47.840085 12796 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:47.851410 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.939426 12647 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:47.853539 14846 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:47.827404 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:47.789381 12721 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:47.944670 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:47.945401 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.945626 12721 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:47.855275 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:47.946300 12645 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:08:47.947369 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.948024 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:47.864665 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 73265.9us
I20260202 15:08:47.948050 12645 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:47.949270 12765 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:47.837039 14883 consensus_meta.cc:287] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.531s user 0.005s sys 0.005s
W20260202 15:08:47.867745 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 83890.1us
I20260202 15:08:47.868121 14950 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.836134 13359 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:47.837481 15067 raft_consensus.cc:3055] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:47.771909 15050 raft_consensus.cc:697] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:47.870764 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1540) took 1670 ms. Trace:
I20260202 15:08:47.819691 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:47.835255 14511 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.015s sys 0.018s
W20260202 15:08:47.880575 12722 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 325) took 3015 ms (client timeout 1821 ms). Trace:
I20260202 15:08:47.870900 14681 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:47.885334 14935 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:47.871125 15038 raft_consensus.cc:697] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:47.859462 15003 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:47.891579 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.840945 15054 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:47.886919 14762 consensus_queue.cc:237] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:47.888046 14924 consensus_queue.cc:237] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.881505 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:47.961120 15067 raft_consensus.cc:740] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:47.925074 15045 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260202 15:08:47.898658 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:47.880743 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:47.881994 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:47.900522 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.901310 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:47.800218 15001 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:47.829944 14847 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:47.905697 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 81207.1us
I20260202 15:08:48.036785 13326 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:47.836724 15062 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.907570 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.907944 13363 raft_consensus.cc:2393] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:47.918828 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 365) took 1710 ms. Trace:
I20260202 15:08:47.920048 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:47.912739 12649 raft_consensus.cc:2376] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7. Re-sending same reply.
W20260202 15:08:47.921902 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:47.923527 12725 raft_consensus.cc:2393] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:47.900553 15061 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:47.936630 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:48.054771 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.939788 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:47.898337 15026 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:47.942054 13358 raft_consensus.cc:1240] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:47.942212 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:48.058000 12644 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:48.001299 14984 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:47.941047 15064 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260202 15:08:47.951481 12690 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:48.064568 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 113146us
I20260202 15:08:48.066016 12690 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:47.829259 14991 raft_consensus.cc:3055] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:47.951582 12689 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:47.919157 14869 raft_consensus.cc:2749] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:47.952905 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:48.070931 12721 raft_consensus.cc:2393] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:47.951591 12688 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:47.954036 15082 log.cc:927] Time spent T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260202 15:08:47.957015 12645 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:47.961756 12765 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:47.961436 12642 rpcz_store.cc:276] 0202 15:08:46.200749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.537067 (+336318us) service_pool.cc:225] Handling call
0202 15:08:47.870558 (+1333491us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54000f91d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:47.870693 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":296040832}
I20260202 15:08:47.962078 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:48.091814 12650 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
W20260202 15:08:48.091850 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 130161us
I20260202 15:08:47.962981 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:47.963668 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:47.941437 15058 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:48.092499 12765 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:47.969399 12722 rpcz_store.cc:269] 0202 15:08:44.865479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:45.181384 (+315905us) service_pool.cc:225] Handling call
0202 15:08:47.880419 (+2699035us) spinlock_profiling.cc:243] Waited 2.34 s on lock 0x7b540033a754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:47.880518 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5386395392}
I20260202 15:08:47.955826 15068 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260202 15:08:47.970711 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:47.959748 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:47.947728 15024 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:47.954824 15027 consensus_queue.cc:260] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.917706 15051 consensus_queue.cc:1059] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260202 15:08:47.995918 12723 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:48.108788 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:48.014683 12647 raft_consensus.cc:2468] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:47.840269 15056 consensus_queue.cc:237] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:47.893882 13362 raft_consensus.cc:2393] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:48.030552 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:48.163702 12723 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:48.164454 12643 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:48.032016 13361 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:48.032883 12798 raft_consensus.cc:2410] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260202 15:08:47.973930 15043 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:48.034041 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:47.918409 14896 consensus_queue.cc:237] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:47.978524 15077 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:48.039826 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.046283 12649 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:48.046741 12795 raft_consensus.cc:2393] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:48.045758 12646 rpcz_store.cc:276] 0202 15:08:46.208662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.792160 (+583498us) service_pool.cc:225] Handling call
0202 15:08:47.870667 (+1078507us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:47.870842 (+ 175us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:47.870864 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:47.870904 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:47.870934 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.870990 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:47.918606 (+ 47616us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000f91d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:47.918760 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2479184128}
I20260202 15:08:48.051578 12800 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:48.055743 12796 raft_consensus.cc:2393] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:47.959084 14883 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.053313 12615 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:48.197746 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1747) took 2286 ms. Trace:
W20260202 15:08:47.862872 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:48.200847 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 147601us
W20260202 15:08:48.201208 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:48.059324 13360 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 7: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:48.062448 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:48.096994 14258 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:48.024156 14960 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:48.025202 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:48.089069 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:48.210662 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:48.211504 12723 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:08:48.212178 12645 raft_consensus.cc:1275] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260202 15:08:48.102193 15019 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:48.101701 14913 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:47.941612 15008 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:48.093786 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:48.094513 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 142961us
W20260202 15:08:48.294277 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:48.307317 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:48.285100 12719 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:48.307941 12722 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
W20260202 15:08:48.308209 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:48.308910 12688 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:48.094588 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 143774us
I20260202 15:08:48.168649 15043 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
W20260202 15:08:48.312063 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:48.095436 12799 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:48.095459 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:48.096146 12650 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:48.098533 12765 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:48.063378 14962 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:48.099900 12641 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:08:48.059652 15026 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:48.071538 14991 raft_consensus.cc:740] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:48.109051 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:48.317025 12717 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:48.162503 12648 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:48.166033 12643 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:48.167055 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.097201 14681 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:48.100723 14930 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:48.197528 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.322247 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.198632 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.198814 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:48.199303 15001 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:48.199697 15039 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:48.203039 12649 raft_consensus.cc:606] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:48.326433 12649 raft_consensus.cc:614] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260202 15:08:48.068496 12644 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:48.327199 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:48.327873 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 260280us
I20260202 15:08:48.328099 12644 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:48.204588 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:48.205236 12718 raft_consensus.cc:2393] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:48.331422 12794 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:48.205500 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:48.332728 13361 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:48.206489 12615 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:48.201411 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1734) took 2509 ms. Trace:
I20260202 15:08:48.167433 13357 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:08:48.165716 12647 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:48.209529 15070 raft_consensus.cc:2764] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260202 15:08:48.106598 13355 raft_consensus.cc:1240] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:48.090629 12690 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:48.340159 12644 raft_consensus.cc:1275] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260202 15:08:48.340929 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 250362us
I20260202 15:08:48.341745 12644 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260202 15:08:48.342237 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 135835us
I20260202 15:08:48.099092 14854 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:48.164999 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.217902 12792 raft_consensus.cc:1240] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:48.218348 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.348320 12763 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:48.348944 12796 raft_consensus.cc:2368] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:48.349601 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.349701 12763 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:48.350483 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.351094 12796 raft_consensus.cc:2393] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 7: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:48.219105 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.351302 12718 raft_consensus.cc:1240] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260202 15:08:48.097091 15038 consensus_queue.cc:237] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.216122 13354 raft_consensus.cc:2393] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 3: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:48.224256 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:48.164253 14846 consensus_queue.cc:1059] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.062s
I20260202 15:08:48.354540 15061 consensus_queue.cc:1048] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:48.225225 15062 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:48.227329 12720 raft_consensus.cc:2393] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:48.355819 12723 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:08:48.352618 15036 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:48.356709 12720 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:48.356979 15036 raft_consensus.cc:697] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:48.210644 15045 consensus_queue.cc:1059] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260202 15:08:48.358144 15045 consensus_queue.cc:1243] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260202 15:08:48.201850 14942 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:48.229295 15092 log.cc:927] Time spent T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260202 15:08:48.359287 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37604 (request call id 1724) took 6825 ms. Trace:
W20260202 15:08:48.230170 15099 log.cc:927] Time spent T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20260202 15:08:48.200850 12726 rpcz_store.cc:276] 0202 15:08:45.911346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.862570 (+1951224us) service_pool.cc:225] Handling call
0202 15:08:47.862989 (+ 419us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:47.863229 (+ 240us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:47.965341 (+102112us) log.cc:844] Serialized 14 byte log entry
0202 15:08:48.197190 (+231849us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:48.197237 (+ 47us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:48.197281 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:48.197336 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:48.197701 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":125131,"prepare.queue_time_us":101302,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":126006,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":175621,"threads_started":2,"wal-append.queue_time_us":203087}
I20260202 15:08:48.215515 15033 consensus_queue.cc:1059] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20260202 15:08:48.233974 15055 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:48.361326 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:48.361438 15068 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:48.362131 13354 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:48.362185 15068 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.240577 15025 raft_consensus.cc:1064] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
W20260202 15:08:48.364298 15025 raft_consensus.cc:1068] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: Unable to promote non-voter 66b8c715b9e6479e9218b08bd3156380: Illegal state: Leader has not yet committed an operation in its own term
I20260202 15:08:48.243860 14851 raft_consensus.cc:993] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab: : Instructing follower 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:48.247190 15002 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:48.250243 12793 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:48.366535 15057 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:48.366832 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.367122 15057 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:48.367694 15057 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:48.368225 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.369421 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.255612 12764 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:48.371310 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 115773us
I20260202 15:08:48.257841 14883 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:48.099459 15058 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:48.091236 15050 consensus_queue.cc:237] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.097657 15067 consensus_queue.cc:260] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.359910 12723 rpcz_store.cc:276] 0202 15:08:41.533689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.356659 (+6822970us) service_pool.cc:225] Handling call
0202 15:08:48.359144 (+ 2485us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:48.377106 15067 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:48.370431 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.264695 14847 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:48.242522 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:48.392835 13362 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:48.200358 14984 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:48.169287 14936 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:48.313647 12722 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:48.315263 12721 raft_consensus.cc:1275] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260202 15:08:48.347090 15001 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260202 15:08:48.320536 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 258124us
I20260202 15:08:48.319252 12688 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:48.216972 15019 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:48.106462 15064 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.209s
I20260202 15:08:48.333716 12724 rpcz_store.cc:276] 0202 15:08:45.692361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.438414 (+1746053us) service_pool.cc:225] Handling call
0202 15:08:47.454102 (+ 15688us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:47.454448 (+ 346us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:47.454470 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:47.464919 (+ 10449us) log.cc:844] Serialized 26 byte log entry
0202 15:08:47.750525 (+285606us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:47.750560 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:47.750620 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:48.201241 (+450621us) raft_consensus.cc:1685] finished
0202 15:08:48.201259 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:48.201364 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":997,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":6911,"thread_start_us":254904,"threads_started":1,"wal-append.queue_time_us":325452}
I20260202 15:08:48.334470 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:48.400735 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 491633us
I20260202 15:08:48.401103 12649 raft_consensus.cc:2393] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 7: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:48.401170 12645 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:48.312287 15043 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:48.335618 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:48.336737 12717 raft_consensus.cc:1275] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260202 15:08:48.338089 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:48.320994 15008 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:48.343318 12690 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:48.319988 14258 raft_consensus.cc:2749] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:48.320436 14681 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:48.213174 15077 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.347918 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:48.380575 12718 raft_consensus.cc:1240] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:48.380692 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.389516 12799 raft_consensus.cc:2410] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260202 15:08:48.354672 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.391160 12797 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:48.414976 12796 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:48.415251 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.416476 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.417137 12797 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:48.392066 13361 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:48.418465 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:48.418730 12689 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:48.419289 13361 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:48.348702 14973 raft_consensus.cc:2749] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:48.395025 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.420819 12795 raft_consensus.cc:2376] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3. Re-sending same reply.
I20260202 15:08:48.395592 12763 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:48.344070 14854 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:48.397378 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 332) took 2184 ms. Trace:
W20260202 15:08:48.315753 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 217303us
I20260202 15:08:48.427295 12722 rpcz_store.cc:276] 0202 15:08:46.213252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.313177 (+2099925us) service_pool.cc:225] Handling call
0202 15:08:48.313470 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:48.397160 (+ 83690us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:48.397330 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:48.399195 13326 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:48.428143 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:48.429008 13326 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:08:48.429100 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:48.399318 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 80131.7us
I20260202 15:08:48.431751 12688 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:48.433207 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.434437 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.402320 12645 raft_consensus.cc:606] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:48.340816 15039 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:48.380019 14991 consensus_queue.cc:260] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:48.402608 15019 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:48.408806 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 66136.7us
I20260202 15:08:48.353611 12791 raft_consensus.cc:2376] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5. Re-sending same reply.
W20260202 15:08:48.412424 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:48.444968 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 92757.9us
W20260202 15:08:48.445214 13328 leader_election.cc:336] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Timed out: RequestConsensusVote RPC to 127.11.46.194:44367 timed out after 1.518s (SENT)
I20260202 15:08:48.445793 13328 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:48.412787 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:48.413177 13362 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:48.446343 12615 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:48.438321 14854 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:48.358296 15036 consensus_queue.cc:237] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.379890 15070 consensus_queue.cc:1059] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260202 15:08:48.438516 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:48.407186 15002 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:48.407284 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:48.440904 13361 raft_consensus.cc:1240] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260202 15:08:48.379634 15042 consensus_queue.cc:1059] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.032s
I20260202 15:08:48.442895 12690 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:48.413580 15001 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:48.422480 14681 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:48.444555 14258 raft_consensus.cc:2804] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:48.450073 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:48.423583 12764 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:48.460918 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 68354.3us
I20260202 15:08:48.452021 12796 raft_consensus.cc:1240] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:48.448967 15062 raft_consensus.cc:2749] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:48.453183 13362 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:08:48.421159 15008 raft_consensus.cc:697] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:48.439786 14973 raft_consensus.cc:1081] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Signalling peer 66b8c715b9e6479e9218b08bd3156380 to start an election
I20260202 15:08:48.458730 12797 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:08:48.491214 12795 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:48.455552 13328 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:48.442139 14847 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:08:48.456454 12798 raft_consensus.cc:2376] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6. Re-sending same reply.
I20260202 15:08:48.442730 14962 raft_consensus.cc:2804] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader election won for term 3
I20260202 15:08:48.448611 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:48.462488 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:48.517627 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:48.431357 14936 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:48.449787 15051 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:48.469273 13361 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:48.469974 13360 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:08:48.454046 14942 consensus_queue.cc:1059] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.041s
W20260202 15:08:48.521307 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 75118.9us
I20260202 15:08:48.522140 12615 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:48.485280 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:48.458787 12645 raft_consensus.cc:614] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260202 15:08:48.492231 12797 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: [4.5-4.5]
W20260202 15:08:48.523346 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:48.523684 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1828) took 1010 ms. Trace:
I20260202 15:08:48.492885 13328 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:48.444237 14865 consensus_queue.cc:1059] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260202 15:08:48.526526 13360 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260202 15:08:48.444120 14913 consensus_queue.cc:1059] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260202 15:08:48.450124 14869 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20260202 15:08:48.449652 14960 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260202 15:08:48.514955 13357 raft_consensus.cc:1240] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260202 15:08:48.455387 15076 consensus_queue.cc:1048] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.059s
I20260202 15:08:48.519153 12765 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:48.524024 12797 rpcz_store.cc:276] 0202 15:08:47.513530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.491510 (+977980us) service_pool.cc:225] Handling call
0202 15:08:48.491747 (+ 237us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:48.523465 (+ 31718us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:48.523640 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:48.465538 15029 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:48.525695 12795 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:48.491986 15001 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260202 15:08:48.529170 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 85683.3us
I20260202 15:08:48.457949 14511 consensus_queue.cc:1048] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
W20260202 15:08:48.502005 14935 raft_consensus.cc:670] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } [suppressed 3 similar messages]
I20260202 15:08:48.491792 15056 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:48.467363 15055 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:48.495313 14258 raft_consensus.cc:697] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:48.513166 15080 log.cc:927] Time spent T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260202 15:08:48.540007 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:48.520658 15054 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260202 15:08:48.544534 12791 raft_consensus.cc:1240] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260202 15:08:48.515802 14847 raft_consensus.cc:697] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:48.551908 12796 raft_consensus.cc:1240] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260202 15:08:48.559036 12720 raft_consensus.cc:1275] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260202 15:08:48.563192 14935 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:48.550923 12726 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:48.554172 13357 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
W20260202 15:08:48.553994 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 100499us
I20260202 15:08:48.545460 14936 raft_consensus.cc:697] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:48.455435 15101 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.087s
I20260202 15:08:48.544201 14973 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:48.565160 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260202 15:08:48.469048 15064 consensus_queue.cc:1059] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260202 15:08:48.514457 15066 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
W20260202 15:08:48.573745 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 130918us
I20260202 15:08:48.570180 15061 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260202 15:08:48.562875 15056 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.631250 12614 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:48.560725 14865 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20260202 15:08:48.575917 15055 raft_consensus.cc:697] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:48.707376 15055 consensus_queue.cc:237] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.572103 15009 consensus_queue.cc:1048] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260202 15:08:48.565789 15008 consensus_queue.cc:237] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.585606 15024 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260202 15:08:48.679224 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:48.679625 12690 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:48.704972 14847 consensus_queue.cc:237] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.712242 12690 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:48.743499 12690 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:48.719590 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:48.744431 12531 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 78ef01280730461580740297403fe090 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
W20260202 15:08:48.729647 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:48.751928 12799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1830) took 1200 ms (client timeout 1525 ms). Trace:
W20260202 15:08:48.752384 12799 rpcz_store.cc:269] 0202 15:08:47.551051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.694623 (+143572us) service_pool.cc:225] Handling call
0202 15:08:48.751733 (+1057110us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54002eb154. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:48.751868 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":636225920}
I20260202 15:08:48.655668 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1826) took 1103 ms. Trace:
W20260202 15:08:48.776063 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 283227us
I20260202 15:08:48.777082 13328 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:48.791129 13354 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:48.792419 13357 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
W20260202 15:08:48.798321 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 246652us
W20260202 15:08:48.796139 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 277079us
I20260202 15:08:48.805004 13328 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:48.775825 12791 rpcz_store.cc:276] 0202 15:08:47.509830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.457749 (+947919us) service_pool.cc:225] Handling call
0202 15:08:48.514939 (+ 57190us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:48.574925 (+ 59986us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:48.613101 (+ 38176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:48.812951 12765 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:48.680279 14973 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:48.833985 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:48.838212 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 316107us
I20260202 15:08:48.742276 15001 consensus_queue.cc:1059] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260202 15:08:48.795657 15070 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:08:48.799252 15117 raft_consensus.cc:3055] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:48.842481 12615 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:48.679917 15116 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.147s
I20260202 15:08:48.808598 12726 raft_consensus.cc:1240] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260202 15:08:48.828572 15064 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:48.827193 15121 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:48.757228 15113 consensus_queue.cc:1059] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.107s
I20260202 15:08:48.841553 15118 raft_consensus.cc:2749] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:48.846511 15070 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:48.852788 15115 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:48.825276 14962 raft_consensus.cc:697] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:48.852438 14865 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:48.794167 14258 consensus_queue.cc:237] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:48.871522 15110 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:48.909773 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 418096us
W20260202 15:08:48.904824 15108 log.cc:927] Time spent T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.363s user 0.001s sys 0.000s
I20260202 15:08:48.871483 15124 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260202 15:08:48.907714 15103 log.cc:927] Time spent T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.362s user 0.001s sys 0.000s
I20260202 15:08:48.851944 15117 raft_consensus.cc:740] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:48.905448 15070 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:48.884446 15061 consensus_queue.cc:1059] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.042s
I20260202 15:08:48.891019 14960 consensus_queue.cc:1059] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260202 15:08:48.844839 14936 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:48.986768 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 187282us
I20260202 15:08:48.939419 15110 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.039708 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 378) took 1273 ms. Trace:
I20260202 15:08:49.019958 15125 raft_consensus.cc:2764] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260202 15:08:49.047282 12642 rpcz_store.cc:276] 0202 15:08:47.766811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.108404 (+341593us) service_pool.cc:225] Handling call
0202 15:08:48.203758 (+ 95354us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:48.204001 (+ 243us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:48.204042 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:48.204456 (+ 414us) log.cc:844] Serialized 26 byte log entry
0202 15:08:48.396135 (+191679us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:48.396169 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:48.396246 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:49.034792 (+638546us) raft_consensus.cc:1685] finished
0202 15:08:49.034817 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:49.039641 (+ 4824us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":137179,"threads_started":1,"wal-append.queue_time_us":197433}
W20260202 15:08:49.033425 15120 log.cc:927] Time spent T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
W20260202 15:08:49.019057 15042 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.000s sys 0.012s
W20260202 15:08:49.052948 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 421729us
I20260202 15:08:49.025624 14865 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:49.071606 15077 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:49.071879 15056 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:48.990298 15130 consensus_queue.cc:1048] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.111s
I20260202 15:08:49.077025 12726 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260202 15:08:49.077919 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 206722us
W20260202 15:08:49.076465 12650 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.001s sys 0.017s
W20260202 15:08:49.054388 15042 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.000s sys 0.013s
W20260202 15:08:49.080847 12650 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.002s sys 0.017s
W20260202 15:08:49.091780 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 155984us
I20260202 15:08:49.056756 15055 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:49.076328 14942 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.000s sys 0.018s
I20260202 15:08:49.091346 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1570) took 1174 ms. Trace:
I20260202 15:08:49.117606 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:49.094547 12763 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:49.096901 15043 consensus_meta.cc:287] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.676s user 0.007s sys 0.000s
I20260202 15:08:49.104910 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:49.122239 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:49.122543 13328 leader_election.cc:400] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:49.153740 13328 leader_election.cc:403] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260202 15:08:49.154764 13328 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:49.155905 13328 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:49.123529 12649 raft_consensus.cc:1240] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260202 15:08:49.117851 12650 rpcz_store.cc:276] 0202 15:08:47.916857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.313711 (+396854us) service_pool.cc:225] Handling call
0202 15:08:48.313915 (+ 204us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:48.314074 (+ 159us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:08:48.449073 (+134999us) log.cc:844] Serialized 14 byte log entry
0202 15:08:49.091065 (+641992us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:49.091119 (+ 54us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:49.091161 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:49.091215 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:49.091301 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37294,"prepare.queue_time_us":227960,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":3503,"raft.queue_time_us":206755,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":36759,"thread_start_us":776458,"threads_started":2}
I20260202 15:08:49.090523 15038 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:49.080113 14973 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:49.119741 14930 consensus_meta.cc:287] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.795s user 0.010s sys 0.000s
I20260202 15:08:49.079851 15136 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:49.158567 12650 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260202 15:08:49.150959 14942 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.000s sys 0.019s
I20260202 15:08:49.075901 14865 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.181493 12648 raft_consensus.cc:1275] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260202 15:08:49.186790 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:49.152464 15076 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:49.052940 14962 consensus_queue.cc:237] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.160542 15077 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:49.166010 14960 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:49.164181 15038 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:49.163909 15121 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:08:49.204794 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 590) took 1414 ms. Trace:
I20260202 15:08:49.211575 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1478) took 1668 ms. Trace:
I20260202 15:08:49.212210 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 589) took 1426 ms. Trace:
I20260202 15:08:49.225162 13363 rpcz_store.cc:276] 0202 15:08:47.785313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.785454 (+ 141us) service_pool.cc:225] Handling call
0202 15:08:49.205956 (+1420502us) spinlock_profiling.cc:243] Waited 49.9 ms on lock 0x7b54005ec4d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:49.212141 (+ 6185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114832128}
I20260202 15:08:49.224588 12725 rpcz_store.cc:276] 0202 15:08:47.543448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.828785 (+285337us) service_pool.cc:225] Handling call
0202 15:08:49.205037 (+1376252us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b5400144e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:49.211458 (+ 6421us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42121088}
I20260202 15:08:49.182180 14930 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:49.171952 15113 raft_consensus.cc:3055] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
W20260202 15:08:49.216596 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 138125us
I20260202 15:08:49.224452 13358 rpcz_store.cc:276] 0202 15:08:47.790190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.790319 (+ 129us) service_pool.cc:225] Handling call
0202 15:08:47.790554 (+ 235us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:48.057741 (+267187us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:49.204527 (+1146786us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54005131d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:49.204728 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":347955968}
I20260202 15:08:49.251353 12763 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:49.240679 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 160346us
I20260202 15:08:49.152231 15055 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.158542 15043 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.167768 15042 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260202 15:08:49.061820 15117 consensus_queue.cc:260] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.228259 15038 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:49.255035 15113 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
W20260202 15:08:49.281404 15120 log.cc:927] Time spent T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260202 15:08:49.259433 15121 raft_consensus.cc:697] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:49.177117 15143 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.117s
W20260202 15:08:49.291209 15026 consensus_meta.cc:287] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.965s user 0.011s sys 0.000s
I20260202 15:08:49.189581 15136 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:49.267081 15056 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260202 15:08:49.250587 14960 raft_consensus.cc:697] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:49.253377 15141 consensus_queue.cc:1059] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
W20260202 15:08:49.336645 15133 log.cc:927] Time spent T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.285s user 0.000s sys 0.001s
W20260202 15:08:49.334686 15057 consensus_meta.cc:287] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.966s user 0.011s sys 0.001s
I20260202 15:08:49.337389 15110 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:49.391886 15113 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:49.401458 15138 log.cc:927] Time spent T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20260202 15:08:49.401206 15145 consensus_queue.cc:1059] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
W20260202 15:08:49.404553 15001 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.008s sys 0.015s
W20260202 15:08:49.414220 15146 log.cc:927] Time spent T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
W20260202 15:08:49.414259 15019 consensus_meta.cc:287] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.961s user 0.008s sys 0.002s
I20260202 15:08:49.414842 15043 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:49.410116 15001 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s user 0.008s sys 0.016s
I20260202 15:08:49.401947 15121 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.414954 15019 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:49.418500 15153 consensus_queue.cc:1059] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260202 15:08:49.420770 15110 consensus_queue.cc:1059] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20260202 15:08:49.424958 15147 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:49.425160 15139 log.cc:927] Time spent T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.240s user 0.001s sys 0.001s
I20260202 15:08:49.425761 15110 raft_consensus.cc:3055] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:49.426177 15110 raft_consensus.cc:740] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:49.427229 15136 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:49.427287 15110 consensus_queue.cc:260] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:49.429014 15137 log.cc:927] Time spent T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260202 15:08:49.429334 15110 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:49.431528 15153 raft_consensus.cc:3055] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:49.432091 15153 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:49.433564 15153 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.435380 15153 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:49.443461 15134 log.cc:927] Time spent T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
W20260202 15:08:49.411340 15141 consensus_queue.cc:1175] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "66b8c715b9e6479e9218b08bd3156380" responder_term: 5 status { last_received { term: 5 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 5 index: 6 } }
I20260202 15:08:49.414969 15154 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:49.449779 15067 consensus_meta.cc:287] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.072s user 0.012s sys 0.000s
I20260202 15:08:49.421408 15113 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:49.448403 15154 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.457571 15150 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260202 15:08:49.460041 15154 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:49.276235 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:49.513554 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 229059us
I20260202 15:08:49.516420 12615 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:49.518129 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:49.529734 12688 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.003s sys 0.004s
W20260202 15:08:49.530165 12688 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s user 0.003s sys 0.004s
W20260202 15:08:49.533033 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:49.279729 12690 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.012s sys 0.015s
W20260202 15:08:49.536211 12690 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.792s user 0.012s sys 0.015s
W20260202 15:08:49.537022 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1474) took 2029 ms (client timeout 2219 ms). Trace:
W20260202 15:08:49.537576 12719 rpcz_store.cc:269] 0202 15:08:47.507562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.737009 (+229447us) service_pool.cc:225] Handling call
0202 15:08:49.536409 (+1799400us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b5400199e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:49.536707 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":876250368}
W20260202 15:08:49.538407 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 794900us
W20260202 15:08:49.555938 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:49.557966 14865 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:49.405892 15057 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.563441 12719 raft_consensus.cc:1275] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260202 15:08:49.395457 14973 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.178s
W20260202 15:08:49.576916 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 77595us
I20260202 15:08:49.577582 14960 consensus_queue.cc:237] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.555992 15055 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:49.583478 15175 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:49.585789 15175 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.582402 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:49.587330 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:49.588306 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:49.588704 15175 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:49.588589 15173 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:49.589172 12648 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:49.584197 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:49.591241 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 75087.9us
I20260202 15:08:49.402042 15026 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.584199 15155 consensus_queue.cc:1059] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260202 15:08:49.596191 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1509) took 1777 ms. Trace:
I20260202 15:08:49.584722 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:49.597769 12647 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 3: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:49.596794 12791 rpcz_store.cc:276] 0202 15:08:47.819015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.845520 (+1026505us) service_pool.cc:225] Handling call
0202 15:08:48.908408 (+ 62888us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:49.255306 (+346898us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:49.258069 (+ 2763us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:49.277590 (+ 19521us) log.cc:844] Serialized 26 byte log entry
0202 15:08:49.595357 (+317767us) raft_consensus.cc:1645] Marking committed up to 2
0202 15:08:49.595452 (+ 95us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:49.595569 (+ 117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:49.595633 (+ 64us) raft_consensus.cc:1685] finished
0202 15:08:49.595660 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:49.595927 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91793,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":13409,"thread_start_us":542057,"threads_started":2,"wal-append.queue_time_us":182303}
I20260202 15:08:49.596135 12615 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:49.598749 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:49.598969 15119 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.004s sys 0.046s
W20260202 15:08:49.599437 15119 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s user 0.004s sys 0.046s
I20260202 15:08:49.599563 12647 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:08:49.599983 15173 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:49.600597 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:49.600965 15119 raft_consensus.cc:1064] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:08:49.602095 12647 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:49.596191 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1508) took 1777 ms. Trace:
I20260202 15:08:49.604898 15019 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:49.604393 12799 rpcz_store.cc:276] 0202 15:08:47.818799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.753208 (+934409us) service_pool.cc:225] Handling call
0202 15:08:48.753527 (+ 319us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:48.786974 (+ 33447us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:48.787024 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:48.787085 (+ 61us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:48.787135 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:49.595648 (+808513us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b54001ccad4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:49.595711 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:49.595940 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76107648}
I20260202 15:08:49.606182 12724 raft_consensus.cc:1240] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:49.606325 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:49.609493 15002 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:49.609711 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:49.610050 15002 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:49.610823 12647 raft_consensus.cc:1240] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260202 15:08:49.603988 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:49.644034 14913 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.099s user 0.006s sys 0.017s
I20260202 15:08:49.645972 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:49.646869 12764 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:49.647575 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:49.648542 12799 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:49.647666 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1506) took 1829 ms. Trace:
I20260202 15:08:49.615389 12615 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:49.630512 14681 consensus_meta.cc:287] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.160s user 0.013s sys 0.008s
W20260202 15:08:49.629611 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 251712us
I20260202 15:08:49.630484 12688 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:49.644599 14913 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.104s user 0.007s sys 0.018s
I20260202 15:08:49.630754 15027 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:49.658221 14849 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:49.658684 12793 raft_consensus.cc:1240] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:49.659374 15163 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:49.660411 15119 consensus_queue.cc:237] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:49.649796 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 388209us
W20260202 15:08:49.666051 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 62505.6us
I20260202 15:08:49.668007 12690 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:49.666944 12615 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:49.672160 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:49.672593 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:49.672906 15019 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260202 15:08:49.673210 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:49.683015 15042 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:49.683526 15042 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:49.683938 15042 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:49.684674 15035 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:49.682967 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:49.708689 15182 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:49.708985 15174 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:49.709690 15175 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:49.710445 15175 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.711774 15154 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:49.711825 15174 raft_consensus.cc:697] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:49.712908 15175 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:49.712740 15154 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:49.713479 15174 consensus_queue.cc:237] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:49.714535 15178 log.cc:927] Time spent T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260202 15:08:49.720362 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:49.721149 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 54389.5us
I20260202 15:08:49.664364 14681 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.722998 12718 raft_consensus.cc:1275] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:49.723215 12721 raft_consensus.cc:1240] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260202 15:08:49.724967 12725 raft_consensus.cc:1275] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260202 15:08:49.724870 12765 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:49.727946 15159 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:49.728489 15159 raft_consensus.cc:697] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:49.731113 14258 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:49.732156 14258 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.731148 15147 consensus_queue.cc:1048] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.002s
I20260202 15:08:49.734618 14258 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:49.719167 14762 consensus_queue.cc:1059] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
W20260202 15:08:49.738509 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:49.738445 15179 raft_consensus.cc:3055] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:49.738929 15179 raft_consensus.cc:740] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:49.740311 12719 raft_consensus.cc:2393] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:49.740041 15179 consensus_queue.cc:260] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.741753 15179 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:49.686201 15057 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:49.690325 15064 consensus_meta.cc:287] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.724s user 0.009s sys 0.002s
I20260202 15:08:49.750070 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:49.745188 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:49.746699 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:49.717123 15140 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:49.747128 13363 raft_consensus.cc:1275] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260202 15:08:49.748381 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:49.749426 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 119129us
I20260202 15:08:49.730572 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:49.770632 15163 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.771354 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:49.649398 12797 rpcz_store.cc:276] 0202 15:08:47.818330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.539053 (+720723us) service_pool.cc:225] Handling call
0202 15:08:48.539401 (+ 348us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:48.704280 (+164879us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:48.704321 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:48.754589 (+ 50268us) log.cc:844] Serialized 52 byte log entry
0202 15:08:49.002418 (+247829us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:49.188515 (+186097us) log.cc:844] Serialized 14 byte log entry
0202 15:08:49.188891 (+ 376us) log.cc:844] Serialized 14 byte log entry
0202 15:08:49.285416 (+ 96525us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:49.285493 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:49.644824 (+359331us) raft_consensus.cc:1685] finished
0202 15:08:49.647113 (+ 2289us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:49.647452 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":333259,"raft.queue_time_us":435136,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":15208,"spinlock_wait_cycles":10103040,"thread_start_us":557292,"threads_started":4,"wal-append.queue_time_us":372173}
I20260202 15:08:49.761418 15035 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:49.800346 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:49.801661 15154 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:49.765314 15175 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:49.772851 15163 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:49.807310 14762 consensus_queue.cc:1048] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260202 15:08:49.808329 14681 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:49.809038 15026 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:49.809303 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 100852us
I20260202 15:08:49.810312 15019 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:49.810806 15140 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:49.811874 15115 consensus_meta.cc:287] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.879s user 0.007s sys 0.011s
I20260202 15:08:49.812321 15115 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:49.781769 15159 consensus_queue.cc:237] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.785660 15064 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:49.883388 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:49.894862 13356 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
W20260202 15:08:50.186482 15042 consensus_meta.cc:287] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.502s user 0.006s sys 0.005s
I20260202 15:08:49.812906 15140 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:50.317387 14950 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:50.321972 15153 consensus_meta.cc:287] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.886s user 0.005s sys 0.013s
W20260202 15:08:50.322105 15110 consensus_meta.cc:287] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.892s user 0.001s sys 0.017s
I20260202 15:08:50.322353 14915 raft_consensus.cc:3055] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Stepping down as leader of term 2
I20260202 15:08:49.820111 15058 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:49.817600 15054 raft_consensus.cc:3055] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:50.771986 15054 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:49.888722 15175 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:50.323302 13354 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
W20260202 15:08:50.336295 15038 consensus_meta.cc:287] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.943s user 0.009s sys 0.004s
W20260202 15:08:50.325234 15070 consensus_meta.cc:287] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.337s user 0.004s sys 0.012s
W20260202 15:08:50.332052 15111 log.cc:927] Time spent T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 1.754s user 0.001s sys 0.001s
W20260202 15:08:50.330935 15113 consensus_meta.cc:287] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.880s user 0.011s sys 0.001s
I20260202 15:08:49.824927 15019 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:50.065220 14854 consensus_meta.cc:287] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.606s user 0.002s sys 0.013s
I20260202 15:08:51.113629 15127 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.113827 14854 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.114226 15127 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.072379 15051 raft_consensus.cc:3055] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:51.115101 15051 raft_consensus.cc:740] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:49.974022 15035 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:49.973201 15095 raft_consensus.cc:3055] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:51.121379 15035 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:51.121461 15095 raft_consensus.cc:740] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:51.122318 15095 consensus_queue.cc:260] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.123672 15095 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:51.124377 15051 consensus_queue.cc:260] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.125680 15051 raft_consensus.cc:3060] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:50.373208 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.128235 12689 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:50.383744 12791 raft_consensus.cc:1240] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:51.131515 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1511) took 3312 ms. Trace:
I20260202 15:08:51.131515 12544 catalog_manager.cc:5654] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 2 to 4, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 4 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:50.427846 12643 raft_consensus.cc:2410] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260202 15:08:51.132877 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.133425 12719 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:51.133704 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 385) took 1555 ms. Trace:
I20260202 15:08:51.133982 12643 rpcz_store.cc:276] 0202 15:08:49.578595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.583722 (+ 5127us) service_pool.cc:225] Handling call
0202 15:08:51.133534 (+1549812us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b540020e754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.133640 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1923046272}
I20260202 15:08:50.441924 15161 raft_consensus.cc:3055] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:51.134611 12719 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:08:51.134909 15161 raft_consensus.cc:740] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:50.426865 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.135735 12645 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:51.136216 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1584) took 1077 ms. Trace:
I20260202 15:08:50.488197 15039 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:50.525398 15061 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:50.526389 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.137241 15130 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.731s user 0.017s sys 0.016s
I20260202 15:08:51.137377 14865 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:51.137888 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.137535 15061 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.137998 14865 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.138471 13328 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:51.139708 15061 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.139779 13328 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:50.527333 15045 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.141041 15118 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.140761 12544 catalog_manager.cc:5654] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:51.141746 15035 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:50.557960 14762 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:51.142594 15033 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.782s user 0.008s sys 0.019s
I20260202 15:08:50.533933 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34778 (request call id 1442) took 2340 ms. Trace:
I20260202 15:08:50.576378 15068 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.142431 12544 catalog_manager.cc:5654] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:50.624805 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:50.636919 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:50.641119 12797 raft_consensus.cc:1240] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260202 15:08:49.820611 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:50.679018 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 584) took 5068 ms. Trace:
I20260202 15:08:51.143069 12646 rpcz_store.cc:276] 0202 15:08:48.193105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.322179 (+129074us) service_pool.cc:225] Handling call
0202 15:08:50.533602 (+2211423us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b540024e354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed5653b5 00007f24ed55d67a 00007f24f2514ff9 00007f24ebaa6db4 00007f24ebaa6d42 00007f24ebaa6c71 00007f24ebaa6bec 00007f24ebaa5ea2 00007f24eb89e34c 00007f24eb89d786
0202 15:08:50.533811 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":169949568}
W20260202 15:08:50.681037 12794 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1829) took 3148 ms (client timeout 2122 ms). Trace:
I20260202 15:08:50.694590 12795 raft_consensus.cc:1240] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
W20260202 15:08:50.703279 12800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1525) took 2850 ms (client timeout 1918 ms). Trace:
I20260202 15:08:50.706369 12650 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:50.333461 15042 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:50.072642 15008 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.776s
I20260202 15:08:50.714046 12690 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:50.066942 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.149461 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.149631 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1578) took 1570 ms. Trace:
W20260202 15:08:51.149910 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:49.899827 12720 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:50.717257 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:50.729720 15148 log.cc:927] Time spent T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 1.404s user 0.001s sys 0.002s
I20260202 15:08:49.926438 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:50.740135 15165 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:50.743521 14960 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:50.743849 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:50.748257 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 343) took 4305 ms. Trace:
I20260202 15:08:50.748342 15036 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:50.752897 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1487) took 2300 ms. Trace:
I20260202 15:08:50.758033 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1504) took 1038 ms. Trace:
I20260202 15:08:50.346640 14915 raft_consensus.cc:740] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:49.936489 15002 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:50.761436 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1485) took 2309 ms. Trace:
W20260202 15:08:50.769213 14258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.945s user 0.005s sys 0.031s
W20260202 15:08:50.332721 15179 consensus_meta.cc:287] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.591s user 0.011s sys 0.001s
I20260202 15:08:50.060402 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:51.156142 14258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.332s user 0.006s sys 0.032s
I20260202 15:08:50.780356 15175 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:50.781128 13356 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 588) took 2996 ms (client timeout 1795 ms). Trace:
W20260202 15:08:49.935290 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 125109us
I20260202 15:08:50.784374 14991 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:50.788158 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.162091 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1505) took 1280 ms. Trace:
I20260202 15:08:51.162323 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1509) took 1180 ms. Trace:
I20260202 15:08:51.162679 12718 rpcz_store.cc:276] 0202 15:08:49.981565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.994302 (+ 12737us) service_pool.cc:225] Handling call
0202 15:08:51.162271 (+1167969us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816}
I20260202 15:08:51.163244 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1583) took 1145 ms. Trace:
I20260202 15:08:51.162454 12720 rpcz_store.cc:276] 0202 15:08:49.881610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.882629 (+ 1019us) service_pool.cc:225] Handling call
0202 15:08:49.899164 (+ 16535us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.151414 (+1252250us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.162032 (+ 10618us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.163758 12642 rpcz_store.cc:276] 0202 15:08:50.017840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:50.053989 (+ 36149us) service_pool.cc:225] Handling call
0202 15:08:51.163191 (+1109202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.161767 15195 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:51.147122 13355 rpcz_store.cc:276] 0202 15:08:45.610439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.439460 (+829021us) service_pool.cc:225] Handling call
0202 15:08:46.439722 (+ 262us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:48.344596 (+1904874us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:50.678558 (+2333962us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54006aead4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:50.678854 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3747359616}
I20260202 15:08:51.164666 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:51.164947 15195 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:51.165239 12642 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:51.165478 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.165541 15195 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:50.960783 13362 raft_consensus.cc:1240] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: []
I20260202 15:08:51.167764 13362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 597) took 2910 ms. Trace:
I20260202 15:08:51.167852 12643 raft_consensus.cc:1275] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:08:51.168090 13362 rpcz_store.cc:276] 0202 15:08:48.256963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.496489 (+239526us) service_pool.cc:225] Handling call
0202 15:08:48.527861 (+ 31372us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.166935 (+2639074us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.167517 (+ 582us) spinlock_profiling.cc:243] Waited 2.43 s on lock 0x7b54004dc6d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:51.167708 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5594288000}
I20260202 15:08:51.169226 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.169734 13362 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:50.985985 15121 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.170496 15177 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:51.172008 15039 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260202 15:08:50.995190 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 1.21394e+06us
I20260202 15:08:51.173010 12615 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:49.819680 12764 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:51.173933 14883 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.174427 15124 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.174399 14883 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:51.175096 15124 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:51.175540 15009 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.176080 15009 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.176558 14883 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:51.025427 15117 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:51.177093 15124 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:51.177640 12648 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:51.030387 15107 log.cc:927] Time spent T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 2.483s user 0.006s sys 0.000s
I20260202 15:08:51.031809 14502 raft_consensus.cc:3055] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:51.178843 14502 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:51.179538 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.179364 15165 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.179677 14502 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.180357 12718 raft_consensus.cc:2410] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 7 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:51.180681 14502 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:51.181344 15165 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.181602 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.033617 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.054236 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1582) took 1083 ms. Trace:
I20260202 15:08:51.066638 15027 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.067790 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 346) took 2656 ms. Trace:
I20260202 15:08:51.087615 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 352) took 3283 ms. Trace:
I20260202 15:08:51.071136 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 599) took 2361 ms. Trace:
I20260202 15:08:51.076339 13326 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:51.110334 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 606) took 1319 ms. Trace:
I20260202 15:08:51.113162 15019 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:51.113346 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1508) took 1171 ms. Trace:
W20260202 15:08:51.113170 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.117024 13359 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:51.115655 14854 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.117615 15127 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.093997 15106 raft_consensus.cc:3055] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:51.198465 12544 catalog_manager.cc:5654] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 4, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 4 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260202 15:08:51.199177 15106 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:08:51.199414 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 123361us
I20260202 15:08:51.200222 15106 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.200857 13326 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:51.109431 15038 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.201968 12546 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39704 (request call id 240) took 7927 ms. Trace:
I20260202 15:08:51.109356 15070 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.152518 15113 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:50.488605 15116 raft_consensus.cc:3055] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:51.135934 15161 consensus_queue.cc:260] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.136485 12645 rpcz_store.cc:276] 0202 15:08:50.058792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:50.426441 (+367649us) service_pool.cc:225] Handling call
0202 15:08:51.136151 (+709710us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.140339 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.140939 15045 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:50.527949 14962 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:50.165715 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.141510 15118 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.211501 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:51.212204 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.212342 15118 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.212463 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:51.213654 15066 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.214078 15066 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:50.679451 12798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1839) took 2775 ms (client timeout 1689 ms). Trace:
I20260202 15:08:51.147512 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1513) took 3327 ms. Trace:
I20260202 15:08:51.155439 12792 rpcz_store.cc:276] 0202 15:08:46.442404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:46.442799 (+ 395us) service_pool.cc:225] Handling call
0202 15:08:46.466222 (+ 23423us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b5800193298. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:46.466404 (+ 182us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:48.352292 (+1885888us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:50.747868 (+2395576us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b54000cbf54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:50.748101 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4000863872}
I20260202 15:08:51.215036 12797 rpcz_store.cc:276] 0202 15:08:47.819560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.882119 (+2062559us) service_pool.cc:225] Handling call
0202 15:08:50.315974 (+433855us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:51.147276 (+831302us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.147463 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.215072 15124 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.215837 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:51.215984 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:51.215907 15124 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.217166 12717 raft_consensus.cc:2393] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:08:51.147269 12794 rpcz_store.cc:269] 0202 15:08:47.532252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.811709 (+279457us) service_pool.cc:225] Handling call
0202 15:08:50.680720 (+2869011us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x7b540004d654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:50.680879 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1150287744}
I20260202 15:08:51.217550 14962 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.218072 15124 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.218262 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:51.219105 12794 raft_consensus.cc:2393] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:51.219594 14962 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.220467 12794 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:51.221155 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1514) took 3401 ms. Trace:
I20260202 15:08:51.221418 15045 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.142351 14762 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:51.147467 12800 rpcz_store.cc:269] 0202 15:08:47.854057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.326982 (+472925us) service_pool.cc:225] Handling call
0202 15:08:50.702908 (+2375926us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b540036db54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:50.703111 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1708916608}
I20260202 15:08:51.222011 15045 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:51.150002 12650 rpcz_store.cc:276] 0202 15:08:49.579144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.586865 (+ 7721us) service_pool.cc:225] Handling call
0202 15:08:51.149508 (+1562643us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540029a254. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.149592 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2571709184}
I20260202 15:08:51.139786 14865 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.153803 15068 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:51.154248 15155 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.226850 12650 raft_consensus.cc:1240] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:51.226995 14865 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:50.779666 15054 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.227223 15155 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.227905 15054 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:51.228284 15118 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.228741 15118 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.229224 15155 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.216082 15145 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.156539 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1507) took 3338 ms. Trace:
I20260202 15:08:51.155527 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1582) took 1374 ms. Trace:
W20260202 15:08:51.150874 15033 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.790s user 0.008s sys 0.020s
I20260202 15:08:51.161337 14915 consensus_queue.cc:260] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:50.959811 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1827) took 3449 ms. Trace:
I20260202 15:08:50.960480 15024 raft_consensus.cc:3055] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:51.178105 15009 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.230661 15118 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.181493 15036 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.231647 15152 consensus_queue.cc:1059] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:08:51.035408 15029 raft_consensus.cc:3055] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:51.232357 15152 consensus_queue.cc:1243] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 1eca64246cad41379cde4e2f9c3e25d2 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260202 15:08:51.158421 13356 rpcz_store.cc:269] 0202 15:08:47.784788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:47.787104 (+ 2316us) service_pool.cc:225] Handling call
0202 15:08:50.780914 (+2993810us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b54006090d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:50.781017 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4187044224}
I20260202 15:08:51.231585 15024 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:51.233414 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.234114 13356 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:51.233668 15024 consensus_queue.cc:260] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.158725 15035 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.225545 14854 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:51.234895 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.170619 14924 consensus_queue.cc:1048] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260202 15:08:51.191026 12723 rpcz_store.cc:276] 0202 15:08:49.942203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.942378 (+ 175us) service_pool.cc:225] Handling call
0202 15:08:51.019427 (+1077049us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.019722 (+ 295us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:08:51.027377 (+ 7655us) log.cc:844] Serialized 14 byte log entry
0202 15:08:51.112895 (+ 85518us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:51.112938 (+ 43us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:51.112992 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.113047 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:51.113155 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64718,"prepare.queue_time_us":9784,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":65907,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":3456,"thread_start_us":27417,"threads_started":2,"wal-append.queue_time_us":22134}
I20260202 15:08:51.231169 15127 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.236603 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.231434 14915 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 2 FOLLOWER]: Advancing to term 3
I20260202 15:08:51.231352 12796 rpcz_store.cc:276] 0202 15:08:47.510101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.465347 (+955246us) service_pool.cc:225] Handling call
0202 15:08:48.465566 (+ 219us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:48.574543 (+108977us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:50.959357 (+2384814us) spinlock_profiling.cc:243] Waited 86.1 ms on lock 0x7b5400116554. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:50.959659 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":198014336}
I20260202 15:08:51.236611 15127 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.227941 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.239590 15070 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.240224 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:51.240839 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 354) took 1439 ms. Trace:
I20260202 15:08:51.190831 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 600) took 1449 ms. Trace:
I20260202 15:08:51.189240 13354 rpcz_store.cc:276] 0202 15:08:49.786110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.793148 (+ 7038us) service_pool.cc:225] Handling call
0202 15:08:51.105736 (+1312588us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.188706 13357 rpcz_store.cc:276] 0202 15:08:48.709248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.709413 (+ 165us) service_pool.cc:225] Handling call
0202 15:08:48.792198 (+ 82785us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:48.797620 (+ 5422us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.070765 (+2273145us) spinlock_profiling.cc:243] Waited 82.4 ms on lock 0x7b5400610150. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.070979 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":189587328}
I20260202 15:08:51.187523 15195 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:51.187209 15027 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.131940 12791 rpcz_store.cc:276] 0202 15:08:47.819292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.770277 (+1950985us) service_pool.cc:225] Handling call
0202 15:08:49.974479 (+204202us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.129925 (+1155446us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.131286 (+ 1361us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b540036c754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:51.131456 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":940628224}
I20260202 15:08:51.155889 12724 rpcz_store.cc:276] 0202 15:08:48.451521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.451616 (+ 95us) service_pool.cc:225] Handling call
0202 15:08:48.459643 (+ 8027us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:49.606715 (+1147072us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:50.761228 (+1154513us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400018bd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:50.761390 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2635434880}
I20260202 15:08:51.241266 12717 rpcz_store.cc:276] 0202 15:08:49.801089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.825469 (+ 24380us) service_pool.cc:225] Handling call
0202 15:08:51.240695 (+1415226us) spinlock_profiling.cc:243] Waited 65.2 ms on lock 0x7b5400031e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.240791 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":150032256}
I20260202 15:08:51.259441 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:51.154320 12531 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 9fae829b1c574bc795c5a424036329b3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica bbf140958c3b4f339c512a9dc4be2dab is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:51.260149 12531 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 106005us
I20260202 15:08:51.202539 15038 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.240741 12645 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:51.155232 12726 rpcz_store.cc:276] 0202 15:08:49.719601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.721397 (+ 1796us) service_pool.cc:225] Handling call
0202 15:08:49.721746 (+ 349us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:49.762908 (+ 41162us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:49.762942 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:49.783623 (+ 20681us) log.cc:844] Serialized 26 byte log entry
0202 15:08:50.172913 (+389290us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:50.172961 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:50.173041 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:50.757829 (+584788us) raft_consensus.cc:1685] finished
0202 15:08:50.757849 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:50.757974 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148323,"raft.queue_time_us":43530,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":29667,"thread_start_us":353638,"threads_started":1,"wal-append.queue_time_us":530871}
W20260202 15:08:51.202828 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 1.38311e+06us
W20260202 15:08:51.275432 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 561427us
I20260202 15:08:51.204686 15166 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:51.201246 12544 catalog_manager.cc:5654] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 2 to 6, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 6 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260202 15:08:50.599115 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.202322 12546 rpcz_store.cc:276] 0202 15:08:43.279342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:43.286847 (+ 7505us) service_pool.cc:225] Handling call
0202 15:08:51.201750 (+7914903us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:08:47.091299 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:08:47.091783 (+ 484us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:08:47.091818 (+ 35us) write_op.cc:435] Acquired schema lock
0202 15:08:47.091829 (+ 11us) tablet.cc:660] Decoding operations
0202 15:08:47.092731 (+ 902us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:08:47.092786 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0202 15:08:47.092809 (+ 23us) tablet.cc:683] Acquiring locks for 12 operations
0202 15:08:47.093619 (+ 810us) tablet.cc:699] Row locks acquired
0202 15:08:47.093635 (+ 16us) write_op.cc:260] PREPARE: finished
0202 15:08:47.093759 (+ 124us) write_op.cc:270] Start()
0202 15:08:47.093860 (+ 101us) write_op.cc:276] Timestamp: P: 1770044927093715 usec, L: 0
0202 15:08:47.093877 (+ 17us) op_driver.cc:348] REPLICATION: starting
0202 15:08:47.094454 (+ 577us) log.cc:844] Serialized 4935 byte log entry
0202 15:08:47.755648 (+661194us) op_driver.cc:464] REPLICATION: finished
0202 15:08:47.897435 (+141787us) write_op.cc:301] APPLY: starting
0202 15:08:47.907084 (+ 9649us) tablet.cc:1365] starting BulkCheckPresence
0202 15:08:47.999288 (+ 92204us) tablet.cc:1368] finished BulkCheckPresence
0202 15:08:48.018595 (+ 19307us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:08:48.021094 (+ 2499us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:08:48.061288 (+ 40194us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0202 15:08:48.072349 (+ 11061us) write_op.cc:312] APPLY: finished
0202 15:08:48.184349 (+112000us) log.cc:844] Serialized 89 byte log entry
0202 15:08:48.215702 (+ 31353us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:48.247793 (+ 32091us) write_op.cc:454] Released schema lock
0202 15:08:48.247880 (+ 87us) write_op.cc:341] FINISH: Updating metrics
0202 15:08:48.248368 (+ 488us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:48.248409 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2850534,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":111485,"apply.run_cpu_time_us":5866,"apply.run_wall_time_us":368904,"num_ops":12,"prepare.queue_time_us":349918,"prepare.run_cpu_time_us":16030,"prepare.run_wall_time_us":323333,"raft.queue_time_us":29747,"raft.run_cpu_time_us":26566,"raft.run_wall_time_us":57338,"replication_time_us":661607,"thread_start_us":538227,"threads_started":4,"wal-append.queue_time_us":417517,"wal-append.run_cpu_time_us":10016,"wal-append.run_wall_time_us":1730939}]]}
I20260202 15:08:51.217061 15161 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:51.182415 12644 rpcz_store.cc:276] 0202 15:08:49.965380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.972354 (+ 6974us) service_pool.cc:225] Handling call
0202 15:08:49.972718 (+ 364us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:49.973024 (+ 306us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:08:50.183832 (+210808us) log.cc:844] Serialized 14 byte log entry
0202 15:08:51.048495 (+864663us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:51.048555 (+ 60us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:51.048580 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.048622 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:51.048715 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":378409,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":1047,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":351,"thread_start_us":285048,"threads_started":2,"wal-append.queue_time_us":83369}
I20260202 15:08:51.222749 12800 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
W20260202 15:08:51.137651 15130 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.734s user 0.017s sys 0.016s
I20260202 15:08:51.221525 12794 rpcz_store.cc:276] 0202 15:08:47.819880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.219870 (+3399990us) service_pool.cc:225] Handling call
0202 15:08:51.220175 (+ 305us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.220869 (+ 694us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.221057 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.304352 12719 raft_consensus.cc:1240] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260202 15:08:51.214850 12798 rpcz_store.cc:269] 0202 15:08:47.908435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.166799 (+258364us) service_pool.cc:225] Handling call
0202 15:08:50.679110 (+2512311us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54003723d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:50.679249 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":318232960}
I20260202 15:08:51.192193 14681 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:51.227589 14865 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.230156 15145 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.160607 15002 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.232491 15029 raft_consensus.cc:740] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:51.230954 12647 rpcz_store.cc:276] 0202 15:08:49.781366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.785698 (+ 4332us) service_pool.cc:225] Handling call
0202 15:08:51.155456 (+1369758us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.155169 12722 rpcz_store.cc:276] 0202 15:08:48.452085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.453778 (+ 1693us) service_pool.cc:225] Handling call
0202 15:08:48.454056 (+ 278us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:48.467915 (+ 13859us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:48.467952 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:48.467999 (+ 47us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:48.468040 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:50.752485 (+2284445us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b5400016dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:50.752543 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:50.752715 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31390464}
I20260202 15:08:51.238366 12796 raft_consensus.cc:1240] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:51.196300 15173 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:51.187458 12721 rpcz_store.cc:276] 0202 15:08:48.411555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.411690 (+ 135us) service_pool.cc:225] Handling call
0202 15:08:48.438373 (+ 26683us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:49.723607 (+1285234us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.067512 (+1343905us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540009f6d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:51.067731 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2953696128}
I20260202 15:08:51.201261 15106 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:51.197516 12648 raft_consensus.cc:2410] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260202 15:08:51.188304 12793 rpcz_store.cc:276] 0202 15:08:47.804611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.516645 (+712034us) service_pool.cc:225] Handling call
0202 15:08:48.535195 (+ 18550us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:49.707721 (+1172526us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.085842 (+1378121us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003575d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:51.087550 (+ 1708us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2355430656}
I20260202 15:08:51.230852 12795 rpcz_store.cc:276] 0202 15:08:47.818550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.555150 (+736600us) service_pool.cc:225] Handling call
0202 15:08:50.694071 (+2138921us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.151536 (+457465us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.156346 (+ 4810us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b540004d650. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.156480 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4918375168}
I20260202 15:08:51.207168 12641 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:51.277400 12689 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:51.241868 13358 rpcz_store.cc:276] 0202 15:08:49.741419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.745980 (+ 4561us) service_pool.cc:225] Handling call
0202 15:08:51.033408 (+1287428us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea4740dc 00007f24ea473c9b 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:08:51.190784 (+157376us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5996288}
I20260202 15:08:51.182183 14960 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.207451 15116 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:51.305431 12723 raft_consensus.cc:2410] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260202 15:08:51.170598 15121 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.215871 15066 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:51.347787 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 354) took 3525 ms. Trace:
W20260202 15:08:51.342921 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 203244us
I20260202 15:08:51.357820 12798 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:51.339995 12650 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:51.350229 12791 raft_consensus.cc:1240] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:51.346483 12644 raft_consensus.cc:1240] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260202 15:08:51.348693 13354 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:51.400274 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1836) took 3497 ms. Trace:
I20260202 15:08:51.319924 13356 raft_consensus.cc:2410] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:51.366590 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 355) took 3526 ms. Trace:
I20260202 15:08:51.400925 12796 rpcz_store.cc:276] 0202 15:08:47.840108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.237912 (+3397804us) service_pool.cc:225] Handling call
0202 15:08:51.238196 (+ 284us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.366325 (+128129us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.366530 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:51.368472 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 144824us
I20260202 15:08:51.369271 12799 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:51.369410 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1583) took 1549 ms. Trace:
I20260202 15:08:51.244328 13355 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
W20260202 15:08:51.371939 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 199019us
I20260202 15:08:51.409312 12726 raft_consensus.cc:1240] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260202 15:08:51.287114 15113 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:51.545629 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1572) took 3231 ms (client timeout 2887 ms). Trace:
I20260202 15:08:51.566438 12795 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:51.462531 12796 raft_consensus.cc:1240] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:51.572521 14924 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:51.400650 12791 rpcz_store.cc:276] 0202 15:08:47.902786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.285986 (+3383200us) service_pool.cc:225] Handling call
0202 15:08:51.350028 (+ 64042us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.400037 (+ 50009us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.400222 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.573040 14924 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.572753 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 365) took 2344 ms. Trace:
I20260202 15:08:51.575168 12796 rpcz_store.cc:276] 0202 15:08:49.228714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.401735 (+2173021us) service_pool.cc:225] Handling call
0202 15:08:51.451144 (+ 49409us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.572554 (+121410us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.572707 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.469774 12690 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:51.542307 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.461238 14258 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:08:51.581781 14258 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:51.582125 14258 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:51.451373 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.451162 12615 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:51.549142 12799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 361) took 2358 ms (client timeout 2142 ms). Trace:
I20260202 15:08:51.420608 12648 rpcz_store.cc:276] 0202 15:08:49.820210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.820382 (+ 172us) service_pool.cc:225] Handling call
0202 15:08:51.369357 (+1548975us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.551724 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 362) took 2356 ms. Trace:
W20260202 15:08:51.583583 12799 rpcz_store.cc:269] 0202 15:08:49.190580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.655688 (+465108us) service_pool.cc:225] Handling call
0202 15:08:51.548977 (+1893289us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540004c254. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.549060 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3277222144}
I20260202 15:08:51.345263 15145 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.337229 15132 consensus_queue.cc:1048] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.268s
I20260202 15:08:51.467103 15070 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:51.571477 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 51098.8us
I20260202 15:08:51.576584 14924 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.346202 13359 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:51.463851 12724 raft_consensus.cc:1275] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260202 15:08:51.554240 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.589216 12791 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260202 15:08:51.487583 15024 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:51.596449 13362 raft_consensus.cc:2410] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260202 15:08:51.596524 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 370) took 1070 ms. Trace:
I20260202 15:08:51.484416 13360 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260202 15:08:51.569363 12641 rpcz_store.cc:269] 0202 15:08:48.314432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.517206 (+1202774us) service_pool.cc:225] Handling call
0202 15:08:51.545426 (+2028220us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b540007a3d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.545563 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3868465408}
I20260202 15:08:51.596958 12791 rpcz_store.cc:276] 0202 15:08:50.526378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.573180 (+1046802us) service_pool.cc:225] Handling call
0202 15:08:51.573449 (+ 269us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.596300 (+ 22851us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.596456 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.579735 13358 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260202 15:08:51.410171 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 132868us
I20260202 15:08:51.570329 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 363) took 2373 ms. Trace:
I20260202 15:08:51.342562 15036 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:51.496650 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 87545.7us
I20260202 15:08:51.584108 12793 rpcz_store.cc:276] 0202 15:08:49.195790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.364137 (+2168347us) service_pool.cc:225] Handling call
0202 15:08:51.466313 (+102176us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.545062 (+ 78749us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:51.545107 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:51.551422 (+ 6315us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:51.551464 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.551535 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:51.551650 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.529826 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.399214 12800 rpcz_store.cc:276] 0202 15:08:47.821905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.222232 (+3400327us) service_pool.cc:225] Handling call
0202 15:08:51.222556 (+ 324us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.311840 (+ 89284us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.347714 (+ 35874us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.588327 12642 raft_consensus.cc:2410] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260202 15:08:51.338868 14681 raft_consensus.cc:697] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:51.578159 15009 consensus_queue.cc:1059] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260202 15:08:51.598402 12641 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
W20260202 15:08:51.537443 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.341761 14762 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:51.582957 12721 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:51.616736 13328 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:51.629287 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.631749 13354 raft_consensus.cc:2410] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:51.629503 12795 rpcz_store.cc:276] 0202 15:08:49.196674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.364151 (+2167477us) service_pool.cc:225] Handling call
0202 15:08:51.506168 (+142017us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.570140 (+ 63972us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.570287 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.600128 12544 catalog_manager.cc:5654] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 4 to 5, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 5 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
W20260202 15:08:51.647547 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.341413 15042 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.347101 15035 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.370301 15127 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.658212 12648 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260202 15:08:51.465091 15121 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.402663 15038 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:51.553977 14865 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.679952 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.679487 12544 catalog_manager.cc:5654] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:51.680523 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:51.680964 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 49570.5us
I20260202 15:08:51.684870 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 360) took 2486 ms. Trace:
W20260202 15:08:51.463153 12578 debug-util.cc:398] Leaking SignalData structure 0x7b0800363720 after lost signal to thread 11454
I20260202 15:08:51.685362 12798 rpcz_store.cc:276] 0202 15:08:49.175808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.319435 (+2143627us) service_pool.cc:225] Handling call
0202 15:08:51.339927 (+ 20492us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.408878 (+ 68951us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.659977 (+251099us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b54001ccfd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:51.661990 (+ 2013us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40610688}
W20260202 15:08:51.686180 12578 debug-util.cc:398] Leaking SignalData structure 0x7b0800091600 after lost signal to thread 12582
W20260202 15:08:51.686473 12578 debug-util.cc:398] Leaking SignalData structure 0x7b080037f800 after lost signal to thread 12596
I20260202 15:08:51.687661 12648 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260202 15:08:51.690392 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 101294us
W20260202 15:08:51.482800 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.691727 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.601518 12797 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:51.658116 14962 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260202 15:08:51.723315 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 128192us
I20260202 15:08:51.711949 13360 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260202 15:08:51.711207 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 241491us
I20260202 15:08:51.716483 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56714 (request call id 1437) took 3312 ms. Trace:
W20260202 15:08:51.715046 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 264038us
W20260202 15:08:51.711422 12763 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.727041 12796 rpcz_store.cc:276] 0202 15:08:48.421580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.580483 (+3158903us) service_pool.cc:225] Handling call
0202 15:08:51.716387 (+135904us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.719517 12765 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:51.728320 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 368) took 1168 ms. Trace:
I20260202 15:08:51.714882 12649 raft_consensus.cc:3055] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:51.728685 12797 rpcz_store.cc:276] 0202 15:08:50.525070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.215733 (+690663us) service_pool.cc:225] Handling call
0202 15:08:51.693279 (+477546us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:51.679360 15175 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:51.524720 12792 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:51.728672 12645 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:08:51.732344 12794 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260202 15:08:51.651969 15165 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:51.705883 12578 debug-util.cc:398] Leaking SignalData structure 0x7b080033f360 after lost signal to thread 12674
I20260202 15:08:51.733778 12650 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:51.474844 15033 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:51.729240 12649 raft_consensus.cc:740] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
W20260202 15:08:51.715734 15177 consensus_meta.cc:287] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.545s user 0.007s sys 0.000s
I20260202 15:08:51.740706 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.744324 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.523669 15066 consensus_queue.cc:1059] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.172s
I20260202 15:08:51.751461 15132 consensus_queue.cc:1048] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.004s
I20260202 15:08:51.710899 15003 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:51.694938 15035 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.755262 15035 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.757375 15035 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.758556 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.759064 15066 raft_consensus.cc:3055] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:51.759284 13362 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:08:51.739550 12641 raft_consensus.cc:1240] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260202 15:08:51.630419 15070 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.760367 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.760962 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:51.761502 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.761935 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.763024 12641 raft_consensus.cc:2410] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 7 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:51.763399 13356 raft_consensus.cc:1240] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260202 15:08:51.763900 15036 raft_consensus.cc:3055] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:51.764323 15036 raft_consensus.cc:740] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:51.764906 13356 raft_consensus.cc:1275] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:51.765285 15036 consensus_queue.cc:260] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.766990 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:51.767722 13358 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:51.768558 14962 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:51.769076 14962 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:51.769098 12764 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:51.771783 15132 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:51.772198 12764 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:51.772224 15132 raft_consensus.cc:697] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:08:51.773345 13359 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 593) took 3922 ms (client timeout 2206 ms). Trace:
I20260202 15:08:51.773267 15132 consensus_queue.cc:237] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.773698 12764 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:51.773866 13359 rpcz_store.cc:269] 0202 15:08:47.851173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:48.198401 (+347228us) service_pool.cc:225] Handling call
0202 15:08:51.773090 (+3574689us) spinlock_profiling.cc:243] Waited 2.79 s on lock 0x7b54005eddd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.773260 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6416006912}
I20260202 15:08:51.774937 14984 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:51.775242 13361 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:51.775498 15177 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:51.775256 14984 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:51.776044 14984 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:51.776211 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 357) took 2611 ms. Trace:
I20260202 15:08:51.748157 15145 consensus_queue.cc:1059] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260202 15:08:51.653141 15116 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.776877 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 604) took 2033 ms. Trace:
I20260202 15:08:51.474877 15173 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.777540 15145 raft_consensus.cc:3055] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:51.777981 15145 raft_consensus.cc:740] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:51.752947 14762 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:51.753409 15033 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.778859 14762 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:51.759426 15066 raft_consensus.cc:740] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:51.761096 15070 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.724264 15113 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.775411 15124 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:51.780817 12721 raft_consensus.cc:2410] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260202 15:08:51.780131 15066 consensus_queue.cc:260] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.781523 15027 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:51.782068 12721 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:51.782222 15066 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:51.783521 12721 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:51.783718 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.752388 15003 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.784642 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.785355 12650 raft_consensus.cc:2410] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260202 15:08:51.785807 15003 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.785898 15209 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:51.786325 15209 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:51.786607 15209 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:51.748381 15166 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.778863 12725 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:51.710827 13354 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68"
dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:44942
W20260202 15:08:51.754529 14991 consensus_meta.cc:287] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.592s user 0.007s sys 0.001s
I20260202 15:08:51.787469 15166 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.788488 14896 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.788931 14896 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.789486 15166 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.748751 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.790405 12645 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:51.790885 12720 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:51.791169 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.791790 12645 raft_consensus.cc:2393] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:51.792477 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.778620 15173 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:51.793313 14915 consensus_meta.cc:287] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.556s user 0.003s sys 0.004s
I20260202 15:08:51.794160 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.794322 14869 consensus_queue.cc:1059] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260202 15:08:51.794723 12646 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
W20260202 15:08:51.722067 15019 consensus_meta.cc:287] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.524s user 0.005s sys 0.001s
I20260202 15:08:51.793016 14896 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.754542 12649 consensus_queue.cc:260] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.793028 12645 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:51.763806 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.794169 12690 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:51.778800 15145 consensus_queue.cc:260] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.776507 12794 rpcz_store.cc:276] 0202 15:08:49.164758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.318917 (+2154159us) service_pool.cc:225] Handling call
0202 15:08:51.343434 (+ 24517us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:51.732077 (+388643us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54001f3720. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4d6df0 00007f24ed560e40 00007f24ed561eb1 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c
0202 15:08:51.746773 (+ 14696us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.776160 (+ 29387us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":315830144}
I20260202 15:08:51.779559 14762 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:51.687794 15038 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.798496 13354 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
W20260202 15:08:51.794310 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.816444 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.766633 15036 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:51.777304 13361 rpcz_store.cc:276] 0202 15:08:49.742993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.747478 (+ 4485us) service_pool.cc:225] Handling call
0202 15:08:51.774782 (+2027304us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.775656 (+ 874us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:51.776610 (+ 954us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b54005eddd0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.776786 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4661436288}
I20260202 15:08:51.719234 14960 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:51.823680 13354 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:51.824517 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.810662 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:51.827826 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.825714 12646 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
W20260202 15:08:51.813968 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 232645us
I20260202 15:08:51.814241 12765 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:51.747459 12578 debug-util.cc:398] Leaking SignalData structure 0x7b0800230420 after lost signal to thread 12750
I20260202 15:08:51.780136 15033 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:51.835533 12578 debug-util.cc:398] Leaking SignalData structure 0x7b080015a280 after lost signal to thread 12751
W20260202 15:08:51.833037 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 340961us
W20260202 15:08:51.837064 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08003d30c0 after lost signal to thread 12824
W20260202 15:08:51.837493 12578 debug-util.cc:398] Leaking SignalData structure 0x7b0800224880 after lost signal to thread 12825
W20260202 15:08:51.819391 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 68472.1us
I20260202 15:08:51.839190 13328 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:51.821468 12645 raft_consensus.cc:1240] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 4. Current term is 5. Ops: []
W20260202 15:08:51.822641 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.840678 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.841686 12578 debug-util.cc:398] Leaking SignalData structure 0x7b0800322580 after lost signal to thread 13387
W20260202 15:08:51.842057 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08003dee60 after lost signal to thread 13388
W20260202 15:08:51.850736 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08002fdc00 after lost signal to thread 14847
W20260202 15:08:51.851037 12578 debug-util.cc:398] Leaking SignalData structure 0x7b0800250600 after lost signal to thread 14849
W20260202 15:08:51.852001 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08002cb6e0 after lost signal to thread 15022
W20260202 15:08:51.853184 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08001a4680 after lost signal to thread 15056
W20260202 15:08:51.853696 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08001b1d40 after lost signal to thread 15067
W20260202 15:08:51.854034 12578 debug-util.cc:398] Leaking SignalData structure 0x7b0800271b00 after lost signal to thread 15076
W20260202 15:08:51.854322 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08003831a0 after lost signal to thread 15077
W20260202 15:08:51.854601 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08003ce7e0 after lost signal to thread 15082
W20260202 15:08:51.854918 12578 debug-util.cc:398] Leaking SignalData structure 0x7b080032f780 after lost signal to thread 15085
W20260202 15:08:51.855211 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08003bcbe0 after lost signal to thread 15088
W20260202 15:08:51.855729 12578 debug-util.cc:398] Leaking SignalData structure 0x7b0800260860 after lost signal to thread 15101
W20260202 15:08:51.856045 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08003a2da0 after lost signal to thread 15103
W20260202 15:08:51.856411 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08003a2d00 after lost signal to thread 15108
W20260202 15:08:51.856743 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08002606c0 after lost signal to thread 15109
W20260202 15:08:51.857405 12578 debug-util.cc:398] Leaking SignalData structure 0x7b08002524c0 after lost signal to thread 15135
W20260202 15:08:51.857748 12578 debug-util.cc:398] Leaking SignalData structure 0x7b0800260800 after lost signal to thread 15136
W20260202 15:08:51.830547 15161 consensus_meta.cc:287] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.526s user 0.009s sys 0.000s
I20260202 15:08:51.865934 12649 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:51.867988 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 53834us
I20260202 15:08:51.817203 14915 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260202 15:08:51.806612 15019 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.815701 15166 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260202 15:08:51.483285 15029 consensus_queue.cc:260] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:51.878540 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.882280 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.872920 15130 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260202 15:08:51.880420 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 136425us
I20260202 15:08:51.881103 12765 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:51.887985 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.893540 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.826733 15121 consensus_queue.cc:1059] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260202 15:08:51.965234 15121 raft_consensus.cc:3055] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:51.965627 15121 raft_consensus.cc:740] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:51.966601 15121 consensus_queue.cc:260] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.968086 15121 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:51.968176 14991 consensus_queue.cc:1048] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260202 15:08:51.893687 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 148884us
I20260202 15:08:51.895182 15054 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:51.870568 15003 raft_consensus.cc:2749] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260202 15:08:51.880993 12792 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 369) took 1346 ms (client timeout 1547 ms). Trace:
W20260202 15:08:51.972256 12792 rpcz_store.cc:269] 0202 15:08:50.526028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.215596 (+689568us) service_pool.cc:225] Handling call
0202 15:08:51.860182 (+644586us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b5400372dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:51.872017 (+ 11835us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":624251136}
I20260202 15:08:51.899295 12645 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:51.899524 14502 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:51.973372 14502 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:51.973704 14502 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:51.973905 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.974900 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.875538 15145 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:51.899927 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.978960 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:51.979506 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:51.893023 15179 consensus_queue.cc:1059] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
I20260202 15:08:51.893379 15130 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:51.980755 15066 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:51.981110 15130 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:51.981405 15179 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.981348 15066 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.982041 15179 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.982957 15124 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.983418 15124 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:51.902922 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.984717 15029 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:51.985119 15124 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:51.985205 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.978971 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:51.905328 15206 consensus_queue.cc:1059] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260202 15:08:51.908262 13328 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:51.908725 15045 consensus_queue.cc:1059] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:51.912853 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:51.915323 15002 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:51.971313 12615 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:51.916332 15215 log.cc:927] Time spent T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260202 15:08:51.917951 14258 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.918872 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.923106 14984 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.924217 15019 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:51.918272 13354 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.997879 13362 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 7. Ops: []
I20260202 15:08:51.958127 15038 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:51.873368 14681 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:51.898916 12690 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:51.986476 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.986554 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.996551 12722 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:52.002861 13354 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:52.003053 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.003209 12765 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:52.004159 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 105361us
I20260202 15:08:52.004293 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.004487 15206 raft_consensus.cc:3055] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:52.004892 13363 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:52.004323 14915 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:52.005416 14915 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:52.005722 14915 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:52.004921 12690 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:52.007431 12647 raft_consensus.cc:2393] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 8: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
W20260202 15:08:52.008399 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 349929us
I20260202 15:08:52.024268 12643 raft_consensus.cc:2368] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:52.024096 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:51.942221 15209 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.031215 14258 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:52.031386 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:51.995378 15066 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:52.036444 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.037243 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:52.029081 15038 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:52.038954 12722 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:52.004186 15127 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:52.039888 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 131727us
I20260202 15:08:52.040584 12690 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:52.040838 15045 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260202 15:08:52.034720 15026 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:52.041666 15054 consensus_queue.cc:1048] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260202 15:08:52.012320 12688 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:52.042990 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.004882 15206 raft_consensus.cc:740] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:52.043551 12647 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
I20260202 15:08:52.044371 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.044155 15206 consensus_queue.cc:260] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.045197 15206 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:52.045286 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.041818 15026 raft_consensus.cc:697] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:52.046881 14991 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:52.047592 14991 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:52.047443 15026 consensus_queue.cc:237] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.048395 14991 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:52.049252 14984 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:52.026634 15177 consensus_queue.cc:1059] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260202 15:08:52.051007 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:52.051399 15177 raft_consensus.cc:3055] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:52.051491 12717 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:52.051723 15177 raft_consensus.cc:740] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:52.052433 15177 consensus_queue.cc:260] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:52.053372 15177 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:52.054003 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 82813.1us
I20260202 15:08:52.039324 15038 raft_consensus.cc:697] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:52.054663 12615 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:52.002918 15210 raft_consensus.cc:3055] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Stepping down as leader of term 5
W20260202 15:08:52.032670 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 303608us
I20260202 15:08:52.056870 13326 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:52.114548 12649 raft_consensus.cc:1275] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260202 15:08:52.115309 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 59019.8us
I20260202 15:08:52.116472 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:52.117228 12646 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:52.117802 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.119344 12763 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:52.121915 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 83397.5us
I20260202 15:08:52.055364 15173 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader election won for term 3
W20260202 15:08:52.120982 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 64336.1us
I20260202 15:08:52.124826 12689 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:52.046401 14962 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.126106 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1581) took 3003 ms. Trace:
I20260202 15:08:52.058310 15210 raft_consensus.cc:740] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:08:52.135717 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.147462 12717 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:52.148738 12765 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:52.135746 13326 leader_election.cc:400] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:52.135999 14762 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.154778 12689 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:52.023648 15002 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:52.136729 15173 raft_consensus.cc:697] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:52.138657 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.114346 15019 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:52.137576 12649 rpcz_store.cc:276] 0202 15:08:49.122812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:49.161531 (+ 38719us) service_pool.cc:225] Handling call
0202 15:08:49.161843 (+ 312us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:51.679984 (+2518141us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b540013a220. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4d6df0 00007f24ed560e40 00007f24ed561eb1 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c
0202 15:08:52.114977 (+434993us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.126047 (+ 11070us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4759158016}
I20260202 15:08:52.149693 13326 leader_election.cc:403] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260202 15:08:52.148401 15166 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:52.153307 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.120728 15054 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election won for term 7
W20260202 15:08:52.156319 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 101737us
I20260202 15:08:52.160744 13328 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:52.145038 14865 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:52.170215 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:52.168963 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.170742 12650 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
W20260202 15:08:52.169620 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.171252 12649 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:52.170503 12689 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:52.116490 15130 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:52.163609 15039 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:08:52.141222 14960 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260202 15:08:52.140645 14991 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260202 15:08:52.172135 13326 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:52.168184 15166 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:52.173961 15051 consensus_meta.cc:287] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.048s user 0.007s sys 0.005s
I20260202 15:08:52.190762 15195 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:52.191481 13358 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:52.184248 15003 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:52.193603 12614 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:52.241761 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 48241.5us
I20260202 15:08:52.242482 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:52.188258 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 67139.1us
W20260202 15:08:52.212498 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.243814 13357 raft_consensus.cc:1275] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260202 15:08:52.232398 12643 raft_consensus.cc:1240] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:52.193670 12649 raft_consensus.cc:1275] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260202 15:08:52.175045 15054 raft_consensus.cc:697] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:08:52.241019 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.245725 12615 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:52.246277 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.247478 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.248106 13362 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:52.189666 15035 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:52.165927 15033 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260202 15:08:52.246912 13363 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:08:52.168599 15019 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:52.252063 13354 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260202 15:08:52.244262 13357 pending_rounds.cc:85] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Aborting all ops after (but not including) 5
I20260202 15:08:52.192438 15106 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5.
I20260202 15:08:52.239549 14869 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:52.258903 12614 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:52.258014 13354 pending_rounds.cc:85] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Aborting all ops after (but not including) 5
I20260202 15:08:52.263564 13354 pending_rounds.cc:107] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260202 15:08:52.263900 13354 raft_consensus.cc:2889] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:52.196287 15166 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:52.192183 15039 raft_consensus.cc:697] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:52.236191 15165 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:52.255080 15175 raft_consensus.cc:2749] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260202 15:08:52.261341 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 90871.2us
I20260202 15:08:52.238361 15195 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:52.268945 12763 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:52.252072 15161 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:52.234884 14502 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:52.251816 15051 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:52.269402 14851 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260202 15:08:52.258409 13357 pending_rounds.cc:107] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260202 15:08:52.321828 13357 raft_consensus.cc:2889] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:52.255546 15179 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:52.258850 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.283865 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.282152 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.287272 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:52.284197 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.278326 13328 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
W20260202 15:08:52.282192 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:52.299698 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.280867 12614 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:52.293390 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.288848 15161 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:52.184778 15038 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:52.324751 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 55891.7us
I20260202 15:08:52.324280 13358 raft_consensus.cc:1275] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:52.377022 12763 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:52.281299 15209 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:52.189924 15210 consensus_queue.cc:260] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.328059 12718 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:52.291862 15155 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260202 15:08:52.383312 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 119167us
W20260202 15:08:52.384696 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.270388 15113 raft_consensus.cc:2804] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:52.278131 15035 raft_consensus.cc:697] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:52.277818 15095 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:52.255554 15132 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260202 15:08:52.288790 15195 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:52.375164 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 96919.6us
I20260202 15:08:52.360631 13360 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
W20260202 15:08:52.326049 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 141810us
I20260202 15:08:52.267755 15182 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
W20260202 15:08:52.379567 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.385861 13354 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:52.381507 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:52.401043 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.280730 15106 consensus_queue.cc:1048] T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.101s
I20260202 15:08:52.356614 15066 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:52.402016 15175 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:52.404675 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.443197 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.405112 15027 consensus_queue.cc:1059] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.003s
I20260202 15:08:52.412498 12722 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 5
I20260202 15:08:52.386566 15116 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:52.381062 15161 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:52.280810 15226 consensus_queue.cc:1048] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.244s
I20260202 15:08:52.414559 13328 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:52.439600 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 62660.9us
I20260202 15:08:52.398201 15070 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:08:52.291837 15227 consensus_queue.cc:1048] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.205s
I20260202 15:08:52.272507 15173 consensus_queue.cc:237] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.508363 12763 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:52.509497 12763 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:52.509804 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.354666 15051 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.387867 15113 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:52.402947 14962 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:52.410938 15002 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:52.501727 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.501991 12689 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:52.502465 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:52.541721 13354 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:52.410943 14762 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:52.507050 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 100161us
I20260202 15:08:52.439348 15209 raft_consensus.cc:2764] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260202 15:08:52.514389 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:52.537453 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 124083us
I20260202 15:08:52.495352 15118 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:52.510141 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.407332 15095 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:52.552937 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.505683 15070 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:52.508791 15227 consensus_queue.cc:1243] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260202 15:08:52.566267 13362 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:52.569526 14502 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:52.574975 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:52.575402 15206 consensus_meta.cc:287] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.000s
W20260202 15:08:52.546066 15228 log.cc:927] Time spent T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.288s user 0.000s sys 0.001s
I20260202 15:08:52.513816 15045 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:52.541081 15113 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:52.585966 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:52.541776 15066 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:52.563901 15003 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:52.437961 15054 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.563166 15130 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:52.584594 15070 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:52.616628 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:52.592900 15117 consensus_meta.cc:287] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.416s user 0.012s sys 0.000s
I20260202 15:08:52.501348 15175 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.602212 15045 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:52.626489 12717 raft_consensus.cc:1240] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260202 15:08:52.605409 15179 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:52.564836 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.624931 13354 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
W20260202 15:08:52.625866 15246 log.cc:927] Time spent T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20260202 15:08:52.442845 15039 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.627017 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.598552 14960 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
W20260202 15:08:52.607941 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 347165us
I20260202 15:08:52.611877 15253 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:52.652316 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:52.632788 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1594) took 1096 ms. Trace:
I20260202 15:08:52.653025 12648 rpcz_store.cc:276] 0202 15:08:51.536069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.695716 (+159647us) service_pool.cc:225] Handling call
0202 15:08:51.696062 (+ 346us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:51.719911 (+ 23849us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:51.719960 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:51.720531 (+ 571us) log.cc:844] Serialized 26 byte log entry
0202 15:08:52.029537 (+309006us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:52.029605 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.029721 (+ 116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:52.632617 (+602896us) raft_consensus.cc:1685] finished
0202 15:08:52.632635 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:52.632739 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2775,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":103,"thread_start_us":308481,"threads_started":1,"wal-append.queue_time_us":452310}
W20260202 15:08:52.633697 15245 log.cc:927] Time spent T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.246s user 0.000s sys 0.001s
W20260202 15:08:52.634161 15233 log.cc:927] Time spent T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.342s user 0.001s sys 0.000s
I20260202 15:08:52.660048 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:52.635015 15235 log.cc:927] Time spent T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.347s user 0.000s sys 0.001s
I20260202 15:08:52.628978 15045 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:52.636689 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 805) took 1475 ms. Trace:
I20260202 15:08:52.684016 13357 rpcz_store.cc:276] 0202 15:08:51.161474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.242524 (+ 81050us) service_pool.cc:225] Handling call
0202 15:08:51.408679 (+166155us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:52.328495 (+919816us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.636483 (+307988us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b5400483354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:52.636632 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1908920576}
I20260202 15:08:52.638875 12645 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:52.647120 13359 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:52.695811 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:52.653111 13355 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:52.610047 15035 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.614449 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.585459 15118 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:52.631399 15238 log.cc:927] Time spent T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260202 15:08:52.647429 12722 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:52.582918 15251 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260202 15:08:52.706243 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:52.719556 12726 raft_consensus.cc:1240] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 5. Ops: []
W20260202 15:08:52.607847 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 434048us
I20260202 15:08:52.656941 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:52.676946 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.678715 12643 raft_consensus.cc:1240] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: []
I20260202 15:08:52.606981 15019 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:52.679929 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 170382us
W20260202 15:08:52.696156 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 159454us
I20260202 15:08:52.704811 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:52.702103 15234 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.292s user 0.000s sys 0.001s
I20260202 15:08:52.710489 12764 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
W20260202 15:08:52.780186 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 69772.8us
W20260202 15:08:52.780464 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:52.714074 12646 consensus_meta.cc:287] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.586s user 0.009s sys 0.000s
I20260202 15:08:52.782466 12646 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:08:52.783370 15240 log.cc:927] Time spent T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.192s user 0.000s sys 0.001s
I20260202 15:08:52.783339 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.784021 12646 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:08:52.783768 15113 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.785063 12646 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260202 15:08:52.785387 12646 pending_rounds.cc:85] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 7
I20260202 15:08:52.785288 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:52.785562 15113 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:52.785694 12646 pending_rounds.cc:107] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260202 15:08:52.785959 12646 raft_consensus.cc:2889] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:52.785986 12725 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:52.697746 15173 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260202 15:08:52.655803 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:52.787160 15244 log.cc:927] Time spent T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20260202 15:08:52.787398 12717 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:08:52.723023 15145 consensus_meta.cc:287] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.747s user 0.010s sys 0.000s
I20260202 15:08:52.718662 15179 consensus_queue.cc:1048] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260202 15:08:52.642396 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:52.720468 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:52.721071 15252 log.cc:927] Time spent T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
W20260202 15:08:52.744665 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.767488 13355 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260202 15:08:52.769012 15175 leader_election.cc:290] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:52.718519 15253 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260202 15:08:52.771027 12721 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
W20260202 15:08:52.771517 15232 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.321s user 0.000s sys 0.001s
W20260202 15:08:52.774099 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 56827.9us
I20260202 15:08:52.775285 15051 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
W20260202 15:08:52.775918 15241 log.cc:927] Time spent T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.274s user 0.000s sys 0.001s
I20260202 15:08:52.773520 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1526) took 1176 ms. Trace:
I20260202 15:08:52.778231 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1590) took 1592 ms. Trace:
I20260202 15:08:52.778733 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.810458 12642 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:52.715670 13357 raft_consensus.cc:2376] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5. Re-sending same reply.
I20260202 15:08:52.790712 12763 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:52.792284 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:52.812331 12725 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:52.794307 13326 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:52.796906 12650 consensus_meta.cc:287] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.622s user 0.005s sys 0.000s
I20260202 15:08:52.813441 12650 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:08:52.798976 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:52.800346 12717 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260202 15:08:52.802096 12646 raft_consensus.cc:1240] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:52.801038 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1527) took 1203 ms. Trace:
W20260202 15:08:52.779536 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 55741.9us
I20260202 15:08:52.814471 12723 rpcz_store.cc:276] 0202 15:08:51.597847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.597976 (+ 129us) service_pool.cc:225] Handling call
0202 15:08:51.598227 (+ 251us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:51.598546 (+ 319us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:51.598564 (+ 18us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:51.681298 (+ 82734us) log.cc:844] Serialized 26 byte log entry
0202 15:08:52.126306 (+445008us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:52.126338 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.126399 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:52.800885 (+674486us) raft_consensus.cc:1685] finished
0202 15:08:52.800901 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:52.800991 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":31870,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":10104,"thread_start_us":325991,"threads_started":1,"wal-append.queue_time_us":505026}
I20260202 15:08:52.806860 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 660) took 1093 ms. Trace:
I20260202 15:08:52.815119 12688 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:52.810345 13354 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260202 15:08:52.792358 14762 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:52.806934 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:52.815651 13356 rpcz_store.cc:276] 0202 15:08:51.713620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.765632 (+ 52012us) service_pool.cc:225] Handling call
0202 15:08:51.766187 (+ 555us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:51.766668 (+ 481us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:51.797162 (+ 30494us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:51.815197 (+ 18035us) log.cc:844] Serialized 564 byte log entry
0202 15:08:52.088042 (+272845us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:52.088105 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.088184 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:52.806576 (+718392us) raft_consensus.cc:1685] finished
0202 15:08:52.806600 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:52.806760 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":209327,"threads_started":1,"wal-append.queue_time_us":361194}
I20260202 15:08:52.816416 12721 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:52.809952 12644 rpcz_store.cc:276] 0202 15:08:51.185934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.446183 (+260249us) service_pool.cc:225] Handling call
0202 15:08:51.558121 (+111938us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:52.025508 (+467387us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:52.025533 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:52.056468 (+ 30935us) log.cc:844] Serialized 26 byte log entry
0202 15:08:52.288590 (+232122us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:52.288627 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.289049 (+ 422us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:52.777996 (+488947us) raft_consensus.cc:1685] finished
0202 15:08:52.778033 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:52.778171 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10818,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":15771,"thread_start_us":493083,"threads_started":2,"wal-append.queue_time_us":640778}
I20260202 15:08:52.820446 12690 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:52.793654 15206 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:52.822772 12722 raft_consensus.cc:3055] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:52.831549 12722 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:52.809890 12724 rpcz_store.cc:276] 0202 15:08:51.597530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.599112 (+ 1582us) service_pool.cc:225] Handling call
0202 15:08:51.618841 (+ 19729us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:52.032047 (+413206us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:52.032086 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:52.032528 (+ 442us) log.cc:844] Serialized 26 byte log entry
0202 15:08:52.608080 (+575552us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:52.608112 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.608150 (+ 38us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:52.773341 (+165191us) raft_consensus.cc:1685] finished
0202 15:08:52.773360 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:52.773470 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":23121,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":29922,"thread_start_us":844340,"threads_started":2,"wal-append.queue_time_us":577884}
I20260202 15:08:52.829905 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:52.832422 12720 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:52.819221 12688 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:52.832567 12645 raft_consensus.cc:1240] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:52.797662 15019 consensus_queue.cc:1059] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260202 15:08:52.836474 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:52.836632 13328 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:52.884274 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 47733.8us
I20260202 15:08:52.885411 13328 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:52.842195 13326 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:52.853717 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:52.869051 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 78416.2us
W20260202 15:08:52.873533 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:52.882061 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 61808.6us
I20260202 15:08:52.793068 15209 consensus_queue.cc:1059] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260202 15:08:52.891975 12763 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:52.809989 15095 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:52.800449 15255 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.108s
I20260202 15:08:52.904974 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:52.905543 12645 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:52.905679 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:52.911770 13357 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:52.900486 13354 pending_rounds.cc:85] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Aborting all ops after (but not including) 4
I20260202 15:08:52.912537 13354 pending_rounds.cc:107] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260202 15:08:52.912799 13354 raft_consensus.cc:2889] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:52.821537 14762 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:52.834180 12765 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:52.822535 15259 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:52.822510 15226 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260202 15:08:52.915303 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 112649us
I20260202 15:08:52.908179 12722 consensus_queue.cc:260] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:52.941274 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 49406.3us
I20260202 15:08:52.837200 15051 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:08:52.942508 13357 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260202 15:08:52.942691 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 108587us
I20260202 15:08:52.945942 12688 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:52.882007 14681 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:08:52.900426 15130 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:52.956548 12722 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:52.921815 15029 consensus_meta.cc:287] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.937s user 0.008s sys 0.003s
I20260202 15:08:52.986156 15253 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
W20260202 15:08:52.959980 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 45567.3us
I20260202 15:08:52.987104 15209 raft_consensus.cc:3055] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:08:52.987208 12615 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:52.987573 15209 raft_consensus.cc:740] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:08:52.914619 15177 consensus_meta.cc:287] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.861s user 0.007s sys 0.004s
I20260202 15:08:52.988400 15173 raft_consensus.cc:2764] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
W20260202 15:08:52.982661 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 66391.9us
I20260202 15:08:52.905359 15036 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:52.989794 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 43492.7us
I20260202 15:08:52.989634 15036 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:52.940510 15166 consensus_meta.cc:287] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.635s user 0.009s sys 0.000s
I20260202 15:08:52.991434 15166 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.993891 15166 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:52.994074 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:52.995728 15253 raft_consensus.cc:697] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:52.988371 15209 consensus_queue.cc:260] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:52.998855 15209 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:52.999183 15036 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:52.997818 15253 consensus_queue.cc:237] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.002370 12615 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:53.003904 15113 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:53.004377 15113 raft_consensus.cc:697] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:53.005339 15113 consensus_queue.cc:237] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:53.009003 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:53.009897 12792 raft_consensus.cc:2410] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260202 15:08:53.013178 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.013931 12794 raft_consensus.cc:2393] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:53.014919 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.015584 12796 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:53.020709 12722 raft_consensus.cc:2410] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260202 15:08:52.910873 14960 raft_consensus.cc:2749] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:52.909811 15118 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:52.905241 15124 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260202 15:08:53.028209 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:52.916499 14762 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:52.900990 14962 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20260202 15:08:52.829955 15206 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:53.027514 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 142149us
W20260202 15:08:53.031514 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.036245 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1537) took 1078 ms. Trace:
W20260202 15:08:53.038636 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:53.025619 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 183494us
I20260202 15:08:52.945405 15095 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:53.044984 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:52.943648 15259 raft_consensus.cc:697] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:52.954113 15182 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:52.956010 15051 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:53.028599 13355 raft_consensus.cc:1240] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: []
I20260202 15:08:53.030401 15226 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:53.029314 14681 raft_consensus.cc:697] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 8 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:52.985345 15130 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:53.047855 12719 rpcz_store.cc:276] 0202 15:08:51.957399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.957546 (+ 147us) service_pool.cc:225] Handling call
0202 15:08:51.957834 (+ 288us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:51.969924 (+ 12090us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:51.969965 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:51.970451 (+ 486us) log.cc:844] Serialized 26 byte log entry
0202 15:08:52.232862 (+262411us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:52.232894 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.232949 (+ 55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:53.036026 (+803077us) raft_consensus.cc:1685] finished
0202 15:08:53.036053 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:53.036183 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11655,"raft.queue_time_us":13889,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":261258,"threads_started":1,"wal-append.queue_time_us":404595}
I20260202 15:08:53.052575 13328 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:53.055094 13326 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:53.097244 12793 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:53.049799 15095 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:53.022616 15161 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.101935 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.111819 12793 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
W20260202 15:08:53.112064 15264 log.cc:927] Time spent T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
W20260202 15:08:53.112352 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.113456 12795 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:53.112986 15259 consensus_queue.cc:237] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:53.114414 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.115090 12797 raft_consensus.cc:2393] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:53.107344 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.062429 15051 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:53.028733 15117 consensus_queue.cc:1059] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
W20260202 15:08:53.116430 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 63882.9us
I20260202 15:08:53.124356 13328 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:53.076975 15003 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:08:53.050328 15145 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.053s
I20260202 15:08:53.117059 12796 raft_consensus.cc:2410] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:53.063547 15226 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:53.108085 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:53.116618 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 61660.8us
W20260202 15:08:53.131247 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 189176us
W20260202 15:08:53.125686 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.116400 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.066991 12792 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:53.101115 15130 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:52.916672 15249 consensus_queue.cc:1059] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.156s
I20260202 15:08:53.006961 15254 consensus_queue.cc:1059] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:08:53.111690 15095 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:53.129024 14960 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:53.144863 12763 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:53.145681 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:08:53.133710 12794 raft_consensus.cc:2410] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:53.103247 15227 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:53.136981 12615 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:53.116300 15274 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:53.145654 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:53.156121 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.156759 12796 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:53.156906 15268 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:53.132635 15271 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:53.132928 15226 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:53.169646 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:53.153016 12763 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:53.124027 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 133359us
W20260202 15:08:53.172407 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.147608 14960 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:53.145269 15270 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:53.146819 14869 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:53.194797 12763 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:53.153573 15274 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:53.161248 15253 raft_consensus.cc:2804] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:53.157601 15268 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.147071 15117 consensus_queue.cc:1059] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260202 15:08:53.162496 15271 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:53.205298 15182 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:53.158134 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:53.192471 15254 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260202 15:08:53.230119 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.196393 15257 raft_consensus.cc:3055] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Stepping down as leader of term 6
W20260202 15:08:53.203198 15024 consensus_meta.cc:287] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.607s user 0.012s sys 0.000s
W20260202 15:08:53.266687 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.194272 15270 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:53.229425 14915 consensus_meta.cc:287] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.223s user 0.014s sys 0.003s
W20260202 15:08:53.247840 15121 consensus_meta.cc:287] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.279s user 0.011s sys 0.002s
I20260202 15:08:53.203576 15274 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:53.194057 14960 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:53.260169 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 65477us
I20260202 15:08:53.209769 15206 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:53.158385 15070 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.151429 15227 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:53.271485 12649 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:53.267446 15255 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:53.203490 15253 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:53.229620 15271 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:53.284548 15272 log.cc:927] Time spent T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20260202 15:08:53.282315 13361 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:53.284489 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.277701 14915 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.286142 15262 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260202 15:08:53.287905 15280 log.cc:927] Time spent T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20260202 15:08:53.187407 14681 consensus_queue.cc:237] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.313146 15253 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:53.301019 15045 consensus_meta.cc:287] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.595s user 0.007s sys 0.001s
I20260202 15:08:53.301579 15070 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:53.201632 14869 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:53.210000 15276 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:08:53.302845 12648 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20260202 15:08:53.337442 15242 log.cc:927] Time spent T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.939s user 0.023s sys 0.038s
I20260202 15:08:53.308092 12720 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:53.272584 15270 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:53.316598 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.320905 15039 consensus_queue.cc:1048] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260202 15:08:53.321302 12763 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:53.303215 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:53.389189 12650 raft_consensus.cc:606] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:53.325706 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 188777us
I20260202 15:08:53.390084 12615 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:53.327160 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.297186 12721 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8.
I20260202 15:08:53.250007 15257 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:53.359303 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.358469 12649 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:08:53.339107 15161 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:53.395081 15289 log.cc:927] Time spent T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.059s user 0.001s sys 0.001s
W20260202 15:08:53.428193 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 107082us
I20260202 15:08:53.428936 12763 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:53.429111 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 97335.6us
I20260202 15:08:53.314358 15255 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:53.360039 12792 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:08:53.343468 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:53.440505 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.443121 12797 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:53.396252 15039 consensus_queue.cc:1243] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260202 15:08:53.392989 15033 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:53.362480 15276 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:53.438182 14984 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:53.476974 12642 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:08:53.352221 15045 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.474558 12765 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:53.473641 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 44795.6us
W20260202 15:08:53.481117 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.451748 13355 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:53.486907 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:53.489616 12642 raft_consensus.cc:1240] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:53.493849 13358 raft_consensus.cc:1240] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260202 15:08:53.493989 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.495689 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:53.483592 15262 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:53.498593 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:53.511629 12797 raft_consensus.cc:2368] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6.
I20260202 15:08:53.498965 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.476392 15276 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:53.498602 12798 raft_consensus.cc:2410] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260202 15:08:53.392335 15257 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.496018 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.498438 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:53.501937 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.485802 15124 raft_consensus.cc:2749] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:53.488266 15054 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260202 15:08:53.488111 15019 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:53.496408 15266 log.cc:927] Time spent T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.253s user 0.001s sys 0.000s
I20260202 15:08:53.504840 15210 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:53.505878 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 413) took 1248 ms. Trace:
I20260202 15:08:53.508426 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:53.544232 12649 rpcz_store.cc:276] 0202 15:08:52.257898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.261066 (+ 3168us) service_pool.cc:225] Handling call
0202 15:08:53.505701 (+1244635us) spinlock_profiling.cc:243] Waited 911 ms on lock 0x7b540008acd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:53.505811 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2095061376}
W20260202 15:08:53.490345 15273 log.cc:927] Time spent T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260202 15:08:53.485656 15118 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:53.552592 15118 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:53.553547 15292 log.cc:927] Time spent T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260202 15:08:53.488978 15179 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:08:53.518173 15262 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.555182 15179 raft_consensus.cc:697] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:53.557458 15262 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:53.503096 15287 log.cc:927] Time spent T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
W20260202 15:08:53.564772 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.565409 12793 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:53.523257 15019 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.525060 15257 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:53.526154 12796 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:53.476204 15033 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.527284 14991 consensus_queue.cc:1059] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260202 15:08:53.528664 15195 consensus_meta.cc:287] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.089s user 0.009s sys 0.002s
I20260202 15:08:53.529716 15227 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:53.530458 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.514880 15275 consensus_queue.cc:1059] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.223s
I20260202 15:08:53.535851 14960 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:53.536531 15283 log.cc:927] Time spent T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260202 15:08:53.555393 14915 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:53.496620 15182 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:53.522265 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:53.556943 15179 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:53.570122 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:53.575079 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 88237.1us
I20260202 15:08:53.580904 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.663527 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1513) took 2490 ms. Trace:
I20260202 15:08:53.665006 12797 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:53.664465 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.664960 12718 rpcz_store.cc:276] 0202 15:08:51.173285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.180878 (+ 7593us) service_pool.cc:225] Handling call
0202 15:08:51.181182 (+ 304us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:52.382310 (+1201128us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:53.663259 (+1280949us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003007d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:53.663454 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2630230272}
I20260202 15:08:53.666514 12765 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:53.595517 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.667213 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:53.667827 12649 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:53.665037 12799 raft_consensus.cc:2410] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260202 15:08:53.612028 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 120278us
I20260202 15:08:53.670586 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:53.555276 14762 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.568035 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:53.674337 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:53.675859 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.676995 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:53.681325 15095 consensus_meta.cc:287] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.522s user 0.003s sys 0.008s
I20260202 15:08:53.608083 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:53.546223 15268 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:53.564679 15116 consensus_meta.cc:287] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.059s user 0.010s sys 0.002s
I20260202 15:08:53.723357 13362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 611) took 1458 ms. Trace:
I20260202 15:08:53.954106 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 616) took 1551 ms. Trace:
I20260202 15:08:54.183231 15095 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.183514 13358 rpcz_store.cc:276] 0202 15:08:52.402303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.653994 (+251691us) service_pool.cc:225] Handling call
0202 15:08:52.683381 (+ 29387us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:53.494217 (+810836us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:53.953814 (+459597us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b540028f0d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:53.954032 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1863368064}
I20260202 15:08:54.181437 13362 rpcz_store.cc:276] 0202 15:08:52.264790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.273235 (+ 8445us) service_pool.cc:225] Handling call
0202 15:08:53.723200 (+1449965us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400512054. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:53.723288 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":437689088}
W20260202 15:08:53.959942 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 1.13384e+06us
I20260202 15:08:54.187657 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:53.826887 15045 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:53.708511 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.869257 13355 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8.
I20260202 15:08:54.191599 12723 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:54.191203 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:54.151491 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.199683 13355 raft_consensus.cc:1240] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260202 15:08:54.200210 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 626) took 1205 ms. Trace:
W20260202 15:08:53.816665 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.200879 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:53.709802 15130 consensus_meta.cc:287] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.566s user 0.004s sys 0.005s
I20260202 15:08:54.201833 15130 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.202286 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.202839 12721 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:54.203644 15130 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:53.978391 14960 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:53.719914 15003 raft_consensus.cc:3055] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:54.205101 15003 raft_consensus.cc:740] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:53.783150 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.205874 12722 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:08:53.835134 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 623) took 1127 ms. Trace:
I20260202 15:08:53.686570 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.206971 12793 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:54.206547 13356 rpcz_store.cc:276] 0202 15:08:52.707565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.822396 (+114831us) service_pool.cc:225] Handling call
0202 15:08:52.900894 (+ 78498us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:53.082213 (+181319us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:53.082250 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:53.117476 (+ 35226us) log.cc:844] Serialized 26 byte log entry
0202 15:08:53.204135 (+ 86659us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:53.204174 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:53.204229 (+ 55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:53.834948 (+630719us) raft_consensus.cc:1685] finished
0202 15:08:53.834970 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:53.835079 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":85614,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":127,"thread_start_us":245460,"threads_started":2,"wal-append.queue_time_us":223325}
W20260202 15:08:53.955571 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.208326 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.209002 12793 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:54.209052 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:54.209312 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:53.724941 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1598) took 1678 ms. Trace:
I20260202 15:08:53.720384 15155 raft_consensus.cc:3055] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:54.209911 12648 rpcz_store.cc:276] 0202 15:08:52.046506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.661016 (+614510us) service_pool.cc:225] Handling call
0202 15:08:52.661330 (+ 314us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:53.352579 (+691249us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:53.724723 (+372144us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b540008a2d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:53.724886 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1474690816}
I20260202 15:08:53.685036 12800 raft_consensus.cc:2393] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:53.701457 14984 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:54.180531 15033 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.211374 12795 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:54.211655 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.187032 12798 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:54.188800 15225 consensus_queue.cc:1048] T 00b1a15d2d1246e1abe52586ddf512ef P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:53.797735 15274 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:54.212646 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:53.949069 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1594) took 2066 ms. Trace:
I20260202 15:08:54.200588 13358 rpcz_store.cc:276] 0202 15:08:52.995134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.187413 (+1192279us) service_pool.cc:225] Handling call
0202 15:08:54.200151 (+ 12738us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:54.214022 15274 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:53.723731 15054 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.213415 12647 rpcz_store.cc:276] 0202 15:08:51.882215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.046132 (+163917us) service_pool.cc:225] Handling call
0202 15:08:52.046408 (+ 276us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:52.046604 (+ 196us) raft_consensus.cc:1534] Early marking committed up to index 8
0202 15:08:52.116360 (+ 69756us) log.cc:844] Serialized 14 byte log entry
0202 15:08:52.190098 (+ 73738us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b180048baa0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe5da 00007f24ed5696aa
0202 15:08:52.412130 (+222032us) log.cc:844] Serialized 14 byte log entry
0202 15:08:53.948562 (+1536432us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b180048baa0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ed10685c 00007f24ea4942eb 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4ae335 00007f24ed556ce0 00007f24ed569396 00007f24ed579f07
0202 15:08:53.948678 (+ 116us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:53.948744 (+ 66us) raft_consensus.cc:1645] Marking committed up to 8
0202 15:08:53.948801 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:53.948886 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:53.949010 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26933,"prepare.queue_time_us":69986,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":7433,"raft.queue_time_us":354330,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":24325,"spinlock_wait_cycles":17848960,"thread_start_us":409690,"threads_started":2,"wal-append.queue_time_us":447875,"wal-append.run_cpu_time_us":2153,"wal-append.run_wall_time_us":1345518}
W20260202 15:08:53.839159 15247 log.cc:927] Time spent T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 1.236s user 0.004s sys 0.012s
I20260202 15:08:54.215271 12723 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260202 15:08:54.215732 13360 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.011s sys 0.072s
W20260202 15:08:53.951591 15265 log.cc:927] Time spent T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.995s user 0.003s sys 0.000s
I20260202 15:08:54.216073 15304 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
W20260202 15:08:53.976353 15051 consensus_meta.cc:287] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.844s user 0.008s sys 0.001s
I20260202 15:08:54.216602 15210 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.217044 12642 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:54.190227 12718 raft_consensus.cc:1240] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20260202 15:08:54.216099 13360 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.513s user 0.012s sys 0.072s
I20260202 15:08:54.217173 15210 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.218187 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:53.957191 15236 log.cc:927] Time spent T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 1.355s user 0.009s sys 0.000s
I20260202 15:08:54.219153 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1601) took 2160 ms. Trace:
I20260202 15:08:54.219142 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.220248 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1598) took 2497 ms. Trace:
I20260202 15:08:54.219184 15210 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:54.220616 12641 rpcz_store.cc:276] 0202 15:08:51.722853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:51.827507 (+104654us) service_pool.cc:225] Handling call
0202 15:08:51.849933 (+ 22426us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:51.861036 (+ 11103us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:51.861075 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:51.877902 (+ 16827us) log.cc:844] Serialized 26 byte log entry
0202 15:08:52.168353 (+290451us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:52.168386 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.168441 (+ 55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:53.956517 (+1788076us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:54.219991 (+263474us) raft_consensus.cc:1685] finished
0202 15:08:54.220011 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:54.220185 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":302,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":6911,"spinlock_wait_cycles":7157632,"thread_start_us":268802,"threads_started":1,"wal-append.queue_time_us":511550}
I20260202 15:08:54.219667 12643 rpcz_store.cc:276] 0202 15:08:52.059050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.773051 (+714001us) service_pool.cc:225] Handling call
0202 15:08:52.775520 (+ 2469us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:52.775895 (+ 375us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:52.775928 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:52.776315 (+ 387us) log.cc:844] Serialized 26 byte log entry
0202 15:08:52.944031 (+167716us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:52.944078 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:52.944161 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:53.951018 (+1006857us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:54.218892 (+267874us) raft_consensus.cc:1685] finished
0202 15:08:54.218909 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:54.219107 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2157,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":11411,"spinlock_wait_cycles":62446848,"thread_start_us":167330,"threads_started":1,"wal-append.queue_time_us":180097}
W20260202 15:08:53.962035 15271 consensus_meta.cc:287] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.619s user 0.008s sys 0.000s
W20260202 15:08:54.216049 12764 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.46.196:36191 timed out after 1.759s (SENT) [suppressed 15 similar messages]
I20260202 15:08:54.197377 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.223956 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.243155 15304 raft_consensus.cc:697] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:54.244973 15267 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.245456 15267 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.245718 15304 consensus_queue.cc:237] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:54.245172 12723 raft_consensus.cc:2393] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:54.201370 12724 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:54.204082 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.247153 12721 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:54.247417 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.247658 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 676) took 1108 ms. Trace:
I20260202 15:08:54.248189 12723 raft_consensus.cc:2393] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:54.248062 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:53.671514 15182 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.248874 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.207612 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:54.249744 12791 raft_consensus.cc:2393] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:54.249891 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.250877 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.251722 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.252509 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.251932 15145 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.253191 15261 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.253399 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:54.253278 15145 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.200284 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 628) took 1154 ms. Trace:
I20260202 15:08:54.211894 15127 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:54.257897 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:54.258213 15276 consensus_meta.cc:287] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.711s user 0.009s sys 0.000s
I20260202 15:08:54.258038 15127 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.861580 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 360) took 1241 ms. Trace:
I20260202 15:08:54.259090 12722 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 3.
W20260202 15:08:53.955258 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 285044us
I20260202 15:08:54.258965 12726 rpcz_store.cc:276] 0202 15:08:52.619748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.633114 (+ 13366us) service_pool.cc:225] Handling call
0202 15:08:52.714725 (+ 81611us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:52.774520 (+ 59795us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:53.861369 (+1086849us) spinlock_profiling.cc:243] Waited 81.3 ms on lock 0x7b540033bb50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:53.861519 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":187036032}
I20260202 15:08:54.258817 15276 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.259842 15127 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.260130 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.260506 12724 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:54.261263 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:54.253962 15261 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.216802 15051 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:53.723423 15019 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.271816 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:53.943529 15259 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:53.949795 15255 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:54.222975 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.224491 15054 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.221443 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:54.279356 15051 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:08:54.225267 15271 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.188616 12690 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:54.230211 15026 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s user 0.001s sys 0.006s
I20260202 15:08:54.223567 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:53.711308 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 608) took 1463 ms. Trace:
I20260202 15:08:54.284310 13359 rpcz_store.cc:276] 0202 15:08:52.247689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.252399 (+ 4710us) service_pool.cc:225] Handling call
0202 15:08:52.289684 (+ 37285us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:52.698640 (+408956us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:53.711021 (+1012381us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b5400640354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:53.711231 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":728596864}
I20260202 15:08:54.285419 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:54.285738 15054 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:54.286120 13359 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:54.180171 15270 consensus_meta.cc:287] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.779s user 0.010s sys 0.000s
I20260202 15:08:53.696542 15195 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.287002 15270 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:54.287851 13361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 610) took 2023 ms (client timeout 1945 ms). Trace:
I20260202 15:08:54.288496 13363 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:08:54.288235 13361 rpcz_store.cc:269] 0202 15:08:52.264463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.266663 (+ 2200us) service_pool.cc:225] Handling call
0202 15:08:54.287697 (+2021034us) spinlock_profiling.cc:243] Waited 922 ms on lock 0x7b540045e2d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:54.287786 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2120453504}
I20260202 15:08:54.288919 15270 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:54.241843 12649 raft_consensus.cc:1240] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260202 15:08:54.289108 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:54.289587 13363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 609) took 2032 ms (client timeout 1759 ms). Trace:
I20260202 15:08:54.290019 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:54.290262 13358 raft_consensus.cc:1240] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:54.290565 12649 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:08:54.289886 13363 rpcz_store.cc:269] 0202 15:08:52.257584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.263183 (+ 5599us) service_pool.cc:225] Handling call
0202 15:08:54.289426 (+2026243us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b540064ffd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:54.289535 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4434366848}
I20260202 15:08:54.260609 15039 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.291282 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:54.291500 15039 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.288254 15195 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:54.243424 12794 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:54.244223 14960 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.185326 15095 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:54.185995 12792 raft_consensus.cc:2410] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:08:54.254500 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:54.247996 13354 rpcz_store.cc:276] 0202 15:08:53.139507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.239141 (+1099634us) service_pool.cc:225] Handling call
0202 15:08:54.247021 (+ 7880us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:54.247359 (+ 338us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:54.247386 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:54.247426 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:54.247461 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.247514 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:54.247611 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:54.253194 12719 raft_consensus.cc:2410] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260202 15:08:54.210392 15155 raft_consensus.cc:740] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:54.255227 13355 rpcz_store.cc:276] 0202 15:08:53.046349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.199078 (+1152729us) service_pool.cc:225] Handling call
0202 15:08:54.199400 (+ 322us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:54.200081 (+ 681us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.200241 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:54.255708 12723 raft_consensus.cc:2393] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:08:53.749085 15226 consensus_meta.cc:287] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.523s user 0.001s sys 0.009s
I20260202 15:08:53.723748 15118 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.291113 15182 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:53.840931 15002 raft_consensus.cc:3055] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Stepping down as leader of term 5
W20260202 15:08:54.249123 14869 consensus_meta.cc:287] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.855s user 0.006s sys 0.002s
W20260202 15:08:54.078907 15231 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.005s sys 0.025s
I20260202 15:08:54.096446 15309 raft_consensus.cc:3055] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Stepping down as leader of term 4
W20260202 15:08:53.965947 15253 consensus_meta.cc:287] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.602s user 0.005s sys 0.001s
I20260202 15:08:54.267144 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:54.278903 12798 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:08:54.268234 15165 consensus_queue.cc:1059] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260202 15:08:54.265147 15210 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:54.285445 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:54.283831 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 95297.2us
I20260202 15:08:54.242684 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 674) took 1110 ms. Trace:
I20260202 15:08:54.293963 12648 pending_rounds.cc:85] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 2
I20260202 15:08:54.320570 15019 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:08:54.258916 15145 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.318514 13361 raft_consensus.cc:1240] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:54.321164 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.323175 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.318153 15300 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.034s
I20260202 15:08:54.330451 12649 raft_consensus.cc:1275] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260202 15:08:54.331158 12531 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 78ef01280730461580740297403fe090 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica bbf140958c3b4f339c512a9dc4be2dab is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:54.331131 15002 raft_consensus.cc:740] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:54.353683 12690 leader_election.cc:400] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6.
I20260202 15:08:54.354066 12690 leader_election.cc:403] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260202 15:08:54.354934 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:54.356484 12723 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:08:54.333436 15309 raft_consensus.cc:740] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:54.358528 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:54.320995 14960 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.340727 12648 pending_rounds.cc:107] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260202 15:08:54.363543 12648 raft_consensus.cc:2889] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:54.340847 12647 raft_consensus.cc:1240] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:54.335896 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:54.364526 12723 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:54.341813 13356 rpcz_store.cc:276] 0202 15:08:53.131940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.208186 (+1076246us) service_pool.cc:225] Handling call
0202 15:08:54.208585 (+ 399us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:54.208899 (+ 314us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:54.208919 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:54.209638 (+ 719us) log.cc:844] Serialized 564 byte log entry
0202 15:08:54.209760 (+ 122us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:54.209794 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.209849 (+ 55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:54.242520 (+ 32671us) raft_consensus.cc:1685] finished
0202 15:08:54.242537 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:54.242637 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:54.293882 12688 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:54.339449 12724 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8.
I20260202 15:08:54.282367 15259 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:54.344292 12726 raft_consensus.cc:2393] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:54.298197 15255 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260202 15:08:54.346647 12719 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:08:54.205928 15003 consensus_queue.cc:260] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:54.333824 15231 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s user 0.011s sys 0.025s
I20260202 15:08:54.362116 13354 raft_consensus.cc:2393] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:54.374437 15116 consensus_queue.cc:1059] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260202 15:08:54.374768 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:54.374686 15309 consensus_queue.cc:260] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:54.375471 13354 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:54.375797 15309 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:54.329387 12765 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:54.376600 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 47284.3us
I20260202 15:08:54.377214 12765 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:54.341459 15039 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:54.377656 15310 log.cc:927] Time spent T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20260202 15:08:54.378159 15039 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:54.378530 15039 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:54.378842 15039 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:54.379624 15314 log.cc:927] Time spent T 69e93e86b75b464fa4c55fdeb141acea P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260202 15:08:54.351918 15276 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.381851 15276 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:54.382386 15276 raft_consensus.cc:697] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:54.383495 15276 consensus_queue.cc:237] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.329389 15226 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:54.370963 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.372068 12724 raft_consensus.cc:1240] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:54.372362 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.375895 15003 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:54.367192 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:54.336367 14869 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.389114 13354 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:08:54.389401 13356 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:54.336022 15253 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.404711 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 814) took 2022 ms. Trace:
I20260202 15:08:54.407234 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.408156 12800 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:54.414602 12726 raft_consensus.cc:1240] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 8. Ops: [6.7-6.7]
I20260202 15:08:54.443291 13356 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: []
I20260202 15:08:54.368921 15267 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:54.370682 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.446796 15275 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.455611 15275 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.457484 15275 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.459098 12720 raft_consensus.cc:1240] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:54.349421 15210 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.461452 12721 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:54.405038 13360 rpcz_store.cc:276] 0202 15:08:52.382828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.508614 (+125786us) service_pool.cc:225] Handling call
0202 15:08:53.392878 (+884264us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800027698. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:08:53.393065 (+ 187us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:53.393426 (+ 361us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:53.393453 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:53.393853 (+ 400us) log.cc:844] Serialized 26 byte log entry
0202 15:08:53.683272 (+289419us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:54.217785 (+534513us) log.cc:844] Serialized 14 byte log entry
0202 15:08:54.386609 (+168824us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.386666 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:54.386721 (+ 55us) raft_consensus.cc:1685] finished
0202 15:08:54.404547 (+ 17826us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:54.404665 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18136,"prepare.queue_time_us":141781,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":124,"raft.queue_time_us":3361,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":31143,"spinlock_wait_cycles":24403200,"thread_start_us":889832,"threads_started":3,"wal-append.queue_time_us":752713}
W20260202 15:08:54.474201 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 195433us
I20260202 15:08:54.362613 15155 consensus_queue.cc:260] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:54.507603 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 166177us
W20260202 15:08:54.507969 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 291917us
W20260202 15:08:54.518831 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 183297us
I20260202 15:08:54.493207 15267 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.535434 12724 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:54.541276 12690 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:54.542461 12690 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:54.544600 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:54.545475 12724 raft_consensus.cc:2393] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:54.509286 15261 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.555441 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.558915 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1557) took 1128 ms. Trace:
I20260202 15:08:54.559376 12720 rpcz_store.cc:276] 0202 15:08:53.430311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:53.430454 (+ 143us) service_pool.cc:225] Handling call
0202 15:08:53.430690 (+ 236us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:54.459432 (+1028742us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.558621 (+ 99189us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400199e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:54.558838 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2364686592}
I20260202 15:08:54.561280 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:54.564133 15270 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:54.559590 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:54.564834 15270 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.567479 14869 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.568356 15270 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:54.568748 14960 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:54.570022 15253 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.572230 12763 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:54.458798 15002 consensus_queue.cc:260] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:54.596758 12793 raft_consensus.cc:1275] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260202 15:08:54.585350 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.587606 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 677) took 1396 ms. Trace:
I20260202 15:08:54.625624 13361 rpcz_store.cc:276] 0202 15:08:53.190769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.289953 (+1099184us) service_pool.cc:225] Handling call
0202 15:08:54.290174 (+ 221us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:54.348249 (+ 58075us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.587309 (+239060us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b540045d654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:54.587535 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64808192}
W20260202 15:08:54.588531 12796 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1858) took 1424 ms (client timeout 1571 ms). Trace:
I20260202 15:08:54.590210 12799 raft_consensus.cc:1240] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260202 15:08:54.626749 12796 rpcz_store.cc:269] 0202 15:08:53.163739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:53.163969 (+ 230us) service_pool.cc:225] Handling call
0202 15:08:54.588342 (+1424373us) spinlock_profiling.cc:243] Waited 80.5 ms on lock 0x7b54001f7cd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:54.588461 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185141632}
I20260202 15:08:54.592219 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1533) took 1105 ms. Trace:
I20260202 15:08:54.532673 15271 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:54.528972 15027 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260202 15:08:54.604497 12723 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:54.602237 12726 raft_consensus.cc:1217] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[4.3-4.3] Dedup: 4.3->[]
I20260202 15:08:54.632289 12726 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260202 15:08:54.603426 12725 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:54.587455 15297 consensus_queue.cc:1059] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:08:54.574913 15261 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:54.608911 12763 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:54.614678 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.534446 15165 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6.
I20260202 15:08:54.619889 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.545804 15267 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.578997 15155 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:54.603003 15257 consensus_meta.cc:287] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.022s user 0.004s sys 0.006s
I20260202 15:08:54.595441 15226 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:54.596827 15259 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:54.631829 14869 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:54.659535 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 422) took 1363 ms. Trace:
I20260202 15:08:54.642096 15002 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:54.651659 15271 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260202 15:08:54.635972 15177 raft_consensus.cc:3055] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:08:54.601635 15039 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:54.627974 12795 rpcz_store.cc:276] 0202 15:08:53.486320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:53.486433 (+ 113us) service_pool.cc:225] Handling call
0202 15:08:54.592024 (+1105591us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b540004d654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:54.592151 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1601518976}
W20260202 15:08:54.630466 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 336603us
I20260202 15:08:54.659952 12644 rpcz_store.cc:276] 0202 15:08:53.296191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:53.310074 (+ 13883us) service_pool.cc:225] Handling call
0202 15:08:53.326929 (+ 16855us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:53.327154 (+ 225us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:53.327387 (+ 233us) log.cc:844] Serialized 14 byte log entry
0202 15:08:54.659235 (+1331848us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:54.659279 (+ 44us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:54.659319 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.659379 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:54.659480 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1331644,"prepare.queue_time_us":66979,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41}
I20260202 15:08:54.632537 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.640936 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:54.688784 12796 raft_consensus.cc:2393] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 8: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:54.642647 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 420) took 1484 ms. Trace:
I20260202 15:08:54.689184 12646 rpcz_store.cc:276] 0202 15:08:53.158579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:53.158734 (+ 155us) service_pool.cc:225] Handling call
0202 15:08:53.185050 (+ 26316us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:53.185255 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:53.247625 (+ 62370us) log.cc:844] Serialized 14 byte log entry
0202 15:08:54.642311 (+1394686us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:54.642362 (+ 51us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:54.642417 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.642488 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:54.642584 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1391132,"prepare.queue_time_us":154653,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":10072,"raft.queue_time_us":1395025,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":2682,"thread_start_us":45528,"threads_started":1}
I20260202 15:08:54.651626 15166 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:54.653718 12642 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:54.694422 12718 raft_consensus.cc:2410] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:54.584753 15210 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:54.652572 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:54.697427 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 44918.1us
I20260202 15:08:54.661191 14869 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:54.666273 15259 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:54.698175 13328 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:54.698902 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:54.685973 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:54.692899 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 1.30277e+06us
I20260202 15:08:54.661542 15036 raft_consensus.cc:3055] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:54.710911 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.713332 12799 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:54.711120 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.712320 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:54.716480 13355 raft_consensus.cc:3055] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:54.716887 13355 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:54.714272 12795 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
W20260202 15:08:54.709012 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 1.19746e+06us
I20260202 15:08:54.718902 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:54.686491 15177 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:54.719942 13326 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:54.744236 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 46142.6us
I20260202 15:08:54.745636 13328 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:54.783744 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.798923 12615 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:54.703050 15070 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260202 15:08:54.802143 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 56586.7us
I20260202 15:08:54.803628 13328 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:54.803913 12642 raft_consensus.cc:2410] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260202 15:08:54.805500 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:54.805598 15271 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:54.806023 13361 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:08:54.807026 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:54.807178 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 87302us
W20260202 15:08:54.807538 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 97216.8us
I20260202 15:08:54.707510 14869 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:54.708806 15165 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.702890 15270 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:54.808960 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.809584 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.811007 12795 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:08:54.811933 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.697847 15226 consensus_queue.cc:1059] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260202 15:08:54.831691 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:54.832214 12647 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:54.708528 15267 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:54.701607 15326 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260202 15:08:54.821969 13328 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:54.720242 15036 raft_consensus.cc:740] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:54.801626 15035 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:54.822252 15210 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election won for term 7
W20260202 15:08:54.838856 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.811385 15161 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:08:54.836041 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:54.842350 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.836905 12796 raft_consensus.cc:1240] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260202 15:08:54.837168 13328 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:54.824240 15173 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:54.717757 15003 consensus_queue.cc:1048] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
I20260202 15:08:54.827705 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:54.839359 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.839989 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:54.840374 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:54.823657 15257 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:54.828518 14960 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:54.901366 12795 raft_consensus.cc:1275] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260202 15:08:54.845193 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 302739us
I20260202 15:08:54.902974 12690 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:54.871690 12799 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:54.871937 12797 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:54.878065 15036 consensus_queue.cc:260] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.840041 15267 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:54.842154 15177 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.885718 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:54.823333 15165 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.842522 15161 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:54.895423 13359 consensus_meta.cc:287] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.609s user 0.010s sys 0.001s
I20260202 15:08:54.873346 15173 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:54.900347 15316 log.cc:927] Time spent T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
W20260202 15:08:54.880945 15328 log.cc:927] Time spent T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260202 15:08:54.912554 13328 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:54.913920 12688 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:54.958949 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 56018.6us
W20260202 15:08:54.959048 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.900990 14960 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:54.924471 12647 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:54.900607 15027 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260202 15:08:54.954674 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 127385us
I20260202 15:08:54.919755 15267 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:54.957398 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.954030 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:54.959604 12690 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:54.922012 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:54.969158 12644 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:54.874590 14681 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:54.900806 15210 raft_consensus.cc:697] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:54.964732 12615 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:54.962018 15113 consensus_queue.cc:1059] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260202 15:08:54.970114 15267 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:54.961067 15117 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:54.966089 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 52260.9us
I20260202 15:08:54.928776 15161 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:54.972528 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 60135.2us
I20260202 15:08:54.976485 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:58672 (request call id 19) took 1687 ms. Trace:
I20260202 15:08:54.956128 15003 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:54.965330 14960 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:54.956619 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:54.992530 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:55.021155 12650 rpcz_store.cc:276] 0202 15:08:53.288680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:53.303095 (+ 14415us) service_pool.cc:225] Handling call
0202 15:08:54.976117 (+1673022us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540030b720. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4dc550 00007f24ed5592d2 00007f24f251713a 00007f24ebab2934 00007f24ebab28c2 00007f24ebab27f1 00007f24ebab276c 00007f24ebab1a22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:54.976408 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2919063424}
W20260202 15:08:55.022406 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.955423 13359 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
W20260202 15:08:55.023106 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 63564.9us
I20260202 15:08:54.916639 15035 consensus_queue.cc:1059] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260202 15:08:54.930902 15257 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.102460 13362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 625) took 2148 ms. Trace:
I20260202 15:08:55.102607 12690 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:55.102797 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 421) took 1943 ms. Trace:
W20260202 15:08:55.070281 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.030352 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:55.103521 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:54.981987 12615 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.083932 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:55.104146 13360 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:55.026309 13328 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:55.103868 12690 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.084393 12764 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:55.118676 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 92438.9us
I20260202 15:08:54.980329 15330 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260202 15:08:55.120939 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 139033us
I20260202 15:08:55.121105 12690 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:55.121570 12615 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.124065 12615 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:55.124015 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 571316us
I20260202 15:08:54.980450 15038 raft_consensus.cc:3055] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:55.027889 12794 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260202 15:08:54.975250 15325 log.cc:927] Time spent T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260202 15:08:55.137043 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:55.083644 15270 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:55.008538 15177 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:55.139096 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:55.164696 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 43667.1us
I20260202 15:08:55.140051 12799 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:55.103269 12645 rpcz_store.cc:276] 0202 15:08:53.158917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:53.159061 (+ 144us) service_pool.cc:225] Handling call
0202 15:08:53.159342 (+ 281us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:54.332710 (+1173368us) spinlock_profiling.cc:243] Waited 4.47 ms on lock 0x7b1800469540. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ed10685c 00007f24ea4942eb 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4ae335 00007f24ed556ce0 00007f24ed55787b 00007f24ed5622cb
0202 15:08:54.335380 (+ 2670us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:54.335421 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:54.335767 (+ 346us) log.cc:844] Serialized 26 byte log entry
0202 15:08:54.452675 (+116908us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:54.452704 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.452768 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:54.621178 (+168410us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b1800469540. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea42646a 00007f24f3aac5c9 00007f24f14348ef 00007f24ea4b1451 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad
0202 15:08:55.102582 (+481404us) raft_consensus.cc:1685] finished
0202 15:08:55.102600 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:55.102747 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27443,"raft.queue_time_us":32971,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":5910,"spinlock_wait_cycles":57335808,"thread_start_us":111039,"threads_started":2,"wal-append.queue_time_us":336685}
I20260202 15:08:55.026952 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:55.165815 13356 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:55.144100 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:54.972919 15341 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:55.124990 15033 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:55.127300 15175 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:54.986558 15259 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:55.027685 15027 consensus_queue.cc:1048] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.139s
I20260202 15:08:55.173041 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:55.173777 13359 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:55.170151 15341 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:55.125349 15332 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:55.174290 15341 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.174564 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:55.172058 15324 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:55.175271 13359 raft_consensus.cc:2393] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:55.175447 15257 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.164867 15117 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:55.176429 15335 log.cc:927] Time spent T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260202 15:08:55.177750 15117 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.175501 15324 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.179273 15036 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:55.179766 15324 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.020133 15227 raft_consensus.cc:3055] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:55.181463 15227 raft_consensus.cc:740] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:55.182202 15227 consensus_queue.cc:260] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:54.991241 15329 log.cc:927] Time spent T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260202 15:08:55.183236 15173 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:54.983373 15113 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:55.184422 15113 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:55.184872 15113 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:55.184633 15341 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.185400 15297 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:55.186507 15341 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:54.984243 15155 consensus_queue.cc:1059] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.154s
I20260202 15:08:55.187103 15297 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.187888 15155 raft_consensus.cc:3055] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:55.188261 15155 raft_consensus.cc:740] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:55.189007 15297 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:55.084998 15003 raft_consensus.cc:697] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:55.122653 15026 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:55.122916 15249 raft_consensus.cc:2749] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:55.145725 12650 raft_consensus.cc:2410] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260202 15:08:55.170840 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 43585.2us
I20260202 15:08:55.164885 15024 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:55.171538 12690 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:55.148190 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:54.980175 14681 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.171061 15270 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:54.973531 15039 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:54.994544 15318 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:55.185838 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 624) took 2274 ms. Trace:
I20260202 15:08:55.136686 15038 raft_consensus.cc:740] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:55.024976 15271 consensus_queue.cc:1059] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.141s
I20260202 15:08:55.102820 13362 rpcz_store.cc:276] 0202 15:08:52.954477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.187805 (+1233328us) service_pool.cc:225] Handling call
0202 15:08:54.188166 (+ 361us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:54.244388 (+ 56222us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:54.244416 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:54.244797 (+ 381us) log.cc:844] Serialized 26 byte log entry
0202 15:08:54.652857 (+408060us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:54.652961 (+ 104us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.653162 (+ 201us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:55.102283 (+449121us) raft_consensus.cc:1685] finished
0202 15:08:55.102300 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:55.102414 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55845,"raft.queue_time_us":72881,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":12189,"thread_start_us":395638,"threads_started":1,"wal-append.queue_time_us":733883}
I20260202 15:08:54.983707 14869 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.123173 15253 consensus_queue.cc:1059] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260202 15:08:55.166688 15326 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:55.269560 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:54.981679 15210 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:55.273277 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 101792us
I20260202 15:08:55.183266 15227 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:55.202406 15121 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.022s sys 0.019s
I20260202 15:08:55.188939 15155 consensus_queue.cc:260] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.338172 13361 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:55.328385 15195 raft_consensus.cc:3055] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:55.313715 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.385938 12724 raft_consensus.cc:2410] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260202 15:08:55.329499 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.265311 15262 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:55.175623 15304 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:55.340526 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.390663 15262 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.329768 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:55.392354 12726 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:55.330112 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:55.394066 15262 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:55.393144 12650 raft_consensus.cc:606] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:55.394790 12650 raft_consensus.cc:614] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260202 15:08:55.395825 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:55.396399 12650 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:55.262554 15265 log.cc:927] Time spent T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20260202 15:08:55.293263 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.175282 15267 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.321977 12719 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:08:55.344208 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.404237 15267 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.404556 12720 raft_consensus.cc:2368] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8.
I20260202 15:08:55.345137 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 629) took 1916 ms. Trace:
I20260202 15:08:55.263002 15260 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:55.345937 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 261623us
I20260202 15:08:55.405676 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:55.406263 13361 raft_consensus.cc:3060] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:55.282831 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:55.406517 12764 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:55.405745 13363 rpcz_store.cc:276] 0202 15:08:53.428526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.291425 (+862899us) service_pool.cc:225] Handling call
0202 15:08:54.291698 (+ 273us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:54.298627 (+ 6929us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:54.298653 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:54.332013 (+ 33360us) log.cc:844] Serialized 52 byte log entry
0202 15:08:54.640629 (+308616us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:54.640661 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.640726 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:55.344875 (+704149us) raft_consensus.cc:1685] finished
0202 15:08:55.344903 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:55.345060 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9055,"raft.queue_time_us":6670,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":19860,"thread_start_us":288011,"threads_started":1,"wal-append.queue_time_us":486188}
I20260202 15:08:55.408212 12764 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:55.409080 12650 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:55.409302 15297 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:08:55.409842 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:55.410056 12764 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:55.361363 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:55.410586 15297 consensus_queue.cc:1059] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260202 15:08:55.359633 12690 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:55.363865 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:55.411019 12649 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:55.365818 15033 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:55.338790 13357 rpcz_store.cc:276] 0202 15:08:52.911170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:52.946306 (+ 35136us) service_pool.cc:225] Handling call
0202 15:08:53.034856 (+ 88550us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:53.035189 (+ 333us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:53.035223 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:53.043144 (+ 7921us) log.cc:844] Serialized 26 byte log entry
0202 15:08:54.352811 (+1309667us) spinlock_profiling.cc:243] Waited 6.48 ms on lock 0x7b18004e1240. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471 00007f24ed51a73f
0202 15:08:54.452968 (+100157us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:54.453003 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:54.453074 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:55.185560 (+732486us) raft_consensus.cc:1685] finished
0202 15:08:55.185589 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:55.185783 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3126,"raft.queue_time_us":75599,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":10931,"spinlock_wait_cycles":14895488,"thread_start_us":99986,"threads_started":1,"wal-append.queue_time_us":1745338}
I20260202 15:08:55.363394 14869 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:55.387135 12717 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:55.281909 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:55.413183 15251 raft_consensus.cc:3055] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:55.413403 12646 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.413571 13361 raft_consensus.cc:2468] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:55.413641 15251 raft_consensus.cc:740] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:55.272544 15024 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
W20260202 15:08:55.389051 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.405987 15260 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.414763 15251 consensus_queue.cc:260] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.416096 15260 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.332880 13328 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:55.416764 12726 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
W20260202 15:08:55.417025 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 84221us
I20260202 15:08:55.418222 12764 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:55.278396 15095 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:55.277654 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 153624us
I20260202 15:08:55.419224 15260 raft_consensus.cc:2749] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260202 15:08:55.419413 12615 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.420409 15033 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:55.420835 15033 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:55.421260 15033 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:55.422019 12615 leader_election.cc:304] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:55.423067 15117 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:55.423177 12615 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.423484 15117 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:55.423820 15117 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.424206 12615 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:55.424888 15175 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:08:55.425313 15175 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:55.425690 15175 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:55.426369 15095 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.428714 15095 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.416267 15251 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:55.432308 15117 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.435213 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.435904 12719 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:55.436604 15117 leader_election.cc:290] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:55.439468 15309 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:55.440452 15276 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:55.441401 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.442356 12719 raft_consensus.cc:2410] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260202 15:08:55.443164 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:55.410363 12650 raft_consensus.cc:606] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:55.444792 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:55.445564 12794 raft_consensus.cc:2393] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:55.410953 15262 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:55.309834 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:55.416353 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.446882 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.418808 13328 leader_election.cc:400] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8.
W20260202 15:08:55.447664 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.447827 13328 leader_election.cc:403] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260202 15:08:55.448653 15267 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:55.448915 12689 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:55.448958 13328 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:55.449209 15267 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.450022 15271 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 8
I20260202 15:08:55.406817 12723 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:55.456369 12794 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:55.450093 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:55.313856 15347 raft_consensus.cc:2749] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:55.266145 15027 consensus_queue.cc:1059] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.122s
I20260202 15:08:55.445940 15033 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.460901 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:55.460132 12646 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
W20260202 15:08:55.461962 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.462824 12688 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:55.381343 15121 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s user 0.023s sys 0.019s
W20260202 15:08:55.464551 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 105018us
I20260202 15:08:55.471791 12690 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.468103 12688 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:55.469156 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.469502 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:55.380740 15318 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:55.475616 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:55.478168 12690 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:55.414914 15024 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.481464 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:55.483238 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.483537 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:55.483876 12650 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:55.486088 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.479959 12720 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:55.385097 15195 raft_consensus.cc:740] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:55.481534 12690 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:55.488893 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.472410 15116 raft_consensus.cc:2804] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:08:55.492544 12719 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
I20260202 15:08:55.476233 14869 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:55.477336 15326 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:55.500211 15225 raft_consensus.cc:3055] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:08:55.501300 15324 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:55.501612 12725 raft_consensus.cc:2393] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:55.474583 15019 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:08:55.507846 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1614) took 1287 ms. Trace:
I20260202 15:08:55.532811 12642 rpcz_store.cc:276] 0202 15:08:54.220553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.221257 (+ 704us) service_pool.cc:225] Handling call
0202 15:08:55.507651 (+1286394us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b540020d854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:55.507789 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":860700032}
I20260202 15:08:55.534058 12642 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:55.534132 15276 raft_consensus.cc:2749] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:55.534652 12642 raft_consensus.cc:606] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.535013 12642 raft_consensus.cc:614] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260202 15:08:55.310489 15003 consensus_queue.cc:237] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.538942 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:55.539341 15346 log.cc:927] Time spent T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260202 15:08:55.539520 12726 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:55.539619 15033 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:55.541462 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:55.541962 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1617) took 1317 ms. Trace:
I20260202 15:08:55.542376 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:55.471968 15165 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:55.471470 15257 raft_consensus.cc:2764] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
W20260202 15:08:55.493017 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.509708 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1551) took 1805 ms. Trace:
I20260202 15:08:55.544664 12793 rpcz_store.cc:276] 0202 15:08:53.704098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.209533 (+505435us) service_pool.cc:225] Handling call
0202 15:08:54.596368 (+386835us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:54.625122 (+ 28754us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:55.509537 (+884415us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b540004d650. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:55.509656 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":889103744}
I20260202 15:08:55.545744 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:55.493050 15347 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:55.546833 12793 raft_consensus.cc:1240] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260202 15:08:55.492619 15026 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:55.547653 15026 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.477275 15038 consensus_queue.cc:260] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.548739 15038 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:55.549237 15019 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:55.549489 15026 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:55.549572 15019 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:55.499526 12717 raft_consensus.cc:1240] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:55.549969 12765 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:55.550612 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.550953 12765 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:55.551162 12717 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:55.551280 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.538959 15326 raft_consensus.cc:697] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:55.551905 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:55.524794 15225 raft_consensus.cc:740] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:55.552701 15166 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:55.553059 15225 consensus_queue.cc:260] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.553748 15270 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.553956 15225 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:55.510982 15027 consensus_queue.cc:1243] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer 1eca64246cad41379cde4e2f9c3e25d2 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260202 15:08:55.479663 15341 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:55.555243 14681 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:55.556054 15271 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8.
I20260202 15:08:55.525110 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:55.556638 12793 raft_consensus.cc:1275] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260202 15:08:55.556962 12794 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:55.524093 12644 consensus_meta.cc:287] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.554s user 0.007s sys 0.001s
I20260202 15:08:55.557783 12644 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8.
I20260202 15:08:55.558525 12644 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:08:55.558995 12644 raft_consensus.cc:606] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.559279 12644 raft_consensus.cc:614] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260202 15:08:55.464382 15260 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260202 15:08:55.478806 15318 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:55.560772 15161 consensus_meta.cc:287] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.530s user 0.004s sys 0.002s
I20260202 15:08:55.561154 15161 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.553035 15166 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:55.563081 15166 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.563517 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260202 15:08:55.563539 15175 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.531284 14762 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:55.552481 15326 consensus_queue.cc:237] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.567090 15195 consensus_queue.cc:260] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.568127 15195 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:55.529498 12647 consensus_meta.cc:287] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.561s user 0.005s sys 0.001s
I20260202 15:08:55.548202 12795 raft_consensus.cc:2368] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 6.
I20260202 15:08:55.474036 15095 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:55.574579 12794 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:08:55.552407 12717 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.516233 12723 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:55.491366 15117 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:55.556191 15267 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:55.542949 12796 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:55.569890 12647 raft_consensus.cc:2468] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:55.542265 12643 rpcz_store.cc:276] 0202 15:08:54.224210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.224323 (+ 113us) service_pool.cc:225] Handling call
0202 15:08:54.242576 (+ 18253us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:54.718753 (+476177us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:54.718780 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:54.792907 (+ 74127us) log.cc:844] Serialized 26 byte log entry
0202 15:08:55.166224 (+373317us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:55.166267 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:55.166335 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:55.541801 (+375466us) raft_consensus.cc:1685] finished
0202 15:08:55.541820 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:55.541917 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":572136,"raft.queue_time_us":550206,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":5926,"thread_start_us":334628,"threads_started":1,"wal-append.queue_time_us":657253}
I20260202 15:08:55.590068 12647 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260202 15:08:55.572487 12763 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:55.646202 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.655658 15175 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:55.657841 12725 raft_consensus.cc:1275] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260202 15:08:55.513500 14869 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:55.658900 14869 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:55.659020 15117 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.660163 12647 raft_consensus.cc:1275] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260202 15:08:55.660460 12647 pending_rounds.cc:85] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 3
I20260202 15:08:55.660637 12647 pending_rounds.cc:107] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260202 15:08:55.660815 12647 raft_consensus.cc:2889] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260202 15:08:55.662881 15310 log.cc:927] Time spent T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
I20260202 15:08:55.531713 15116 raft_consensus.cc:697] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:08:55.664325 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 84989.9us
I20260202 15:08:55.570307 15324 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:55.664871 15324 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:55.665287 15117 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.666347 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:55.666841 15127 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:08:55.667832 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:55.668160 15127 consensus_queue.cc:1048] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:08:55.668601 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:55.668905 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:55.669175 12793 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:55.669368 12794 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.524654 15304 consensus_queue.cc:1059] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.146s
I20260202 15:08:55.674993 12793 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
W20260202 15:08:55.676136 14960 consensus_meta.cc:287] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.571s user 0.007s sys 0.000s
I20260202 15:08:55.676602 14960 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:55.677793 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.680230 12725 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260202 15:08:55.680612 12763 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.680907 12717 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:55.681490 15262 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:08:55.681829 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:55.682179 15262 raft_consensus.cc:697] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:55.683123 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:55.683447 15262 consensus_queue.cc:237] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.686632 15304 raft_consensus.cc:2804] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:08:55.687067 15276 raft_consensus.cc:3055] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:55.687341 15304 raft_consensus.cc:697] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:55.687628 15276 raft_consensus.cc:740] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:55.688802 15276 consensus_queue.cc:260] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.690007 15161 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.690274 15276 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.553686 15297 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:55.693684 15297 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:55.694124 15297 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:55.697914 12794 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
W20260202 15:08:55.654214 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 80908.6us
I20260202 15:08:55.699688 12765 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:55.700003 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.664057 15116 consensus_queue.cc:237] T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.701805 12690 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.703140 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:55.719463 12690 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:55.723157 14960 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.725998 12793 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260202 15:08:55.556111 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:55.728422 12690 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:55.734946 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 675) took 2600 ms. Trace:
I20260202 15:08:55.735460 13358 rpcz_store.cc:276] 0202 15:08:53.134689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.214097 (+1079408us) service_pool.cc:225] Handling call
0202 15:08:54.290093 (+ 75996us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:54.290610 (+ 517us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:55.734731 (+1444121us) spinlock_profiling.cc:243] Waited 75.7 ms on lock 0x7b540064ffd0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:55.734891 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174177024}
W20260202 15:08:55.732141 12532 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 9fae829b1c574bc795c5a424036329b3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 66b8c715b9e6479e9218b08bd3156380 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:08:55.708694 15327 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:55.685976 15024 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.709348 15095 raft_consensus.cc:697] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:55.728026 14924 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:55.731491 15036 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:08:55.688814 15304 consensus_queue.cc:237] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.791934 15036 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:55.792315 15036 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:55.771440 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:55.793473 13358 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:55.794339 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:55.795250 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:55.795814 13358 raft_consensus.cc:2393] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 8: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:55.731600 15249 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:55.698009 15113 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.797256 15166 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.798390 15113 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.738899 14762 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:55.799065 14762 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.799309 15095 consensus_queue.cc:237] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.801075 14762 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:55.738543 14681 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:08:55.802192 14681 raft_consensus.cc:697] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:55.802985 14681 consensus_queue.cc:237] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.798624 12796 raft_consensus.cc:2368] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5.
I20260202 15:08:55.806794 12646 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:55.811506 13357 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.811764 12643 raft_consensus.cc:1240] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:55.744113 15332 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:55.809731 15267 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:55.816332 15267 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.816856 13359 raft_consensus.cc:1240] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260202 15:08:55.817739 12751 heartbeater.cc:646] Failed to heartbeat to 127.11.46.254:37385 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.46.254:37385 timed out after 15.000s (SENT)
I20260202 15:08:55.818295 13358 raft_consensus.cc:1240] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260202 15:08:55.814858 13362 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260202 15:08:55.815078 12689 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:55.820358 15249 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:55.821056 15267 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:55.820695 15036 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.824754 15024 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.826743 15036 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:55.826965 15177 consensus_meta.cc:287] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.657s user 0.006s sys 0.002s
I20260202 15:08:55.743947 15333 consensus_queue.cc:1059] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20260202 15:08:55.828308 15177 raft_consensus.cc:3055] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:55.828666 15177 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:55.829419 15177 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.829802 15019 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.830384 15177 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:55.835286 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:55.743556 15356 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:08:55.843040 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 369115us
I20260202 15:08:55.772274 15260 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:55.852043 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.853588 12724 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
I20260202 15:08:55.853787 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.854578 12717 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
I20260202 15:08:55.856578 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:55.790081 15127 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:55.744499 15352 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
W20260202 15:08:55.864629 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.862879 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:55.876778 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1778) took 1655 ms (client timeout 1571 ms). Trace:
W20260202 15:08:55.877225 12721 rpcz_store.cc:269] 0202 15:08:54.221543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.247834 (+ 26291us) service_pool.cc:225] Handling call
0202 15:08:55.876562 (+1628728us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54002fe754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:55.876664 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":356332800}
I20260202 15:08:55.859314 15260 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.880298 13362 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:55.883204 15260 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.771534 15327 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.907217 15327 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:55.907558 15155 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:55.909011 15318 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.910305 15341 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:55.911674 14869 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.913944 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:55.914163 12614 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:55.914549 12646 raft_consensus.cc:2393] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 8: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:55.910316 13357 raft_consensus.cc:1275] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:08:55.922765 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:55.926491 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.889923 12717 raft_consensus.cc:2410] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260202 15:08:55.859499 15356 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.902876 15324 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
W20260202 15:08:55.932716 13363 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.032s sys 0.036s
W20260202 15:08:55.932917 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:55.934108 13363 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.525s user 0.032s sys 0.036s
W20260202 15:08:55.942516 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:55.952498 12614 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:55.955216 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:55.956001 12646 raft_consensus.cc:2393] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
W20260202 15:08:55.982256 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:55.982846 12645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s user 0.003s sys 0.016s
I20260202 15:08:55.851372 15297 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:55.984006 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:55.994460 12646 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:55.972070 15161 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:55.995291 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:55.995168 15161 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:55.996021 12646 raft_consensus.cc:2393] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 4: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:55.996773 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:55.997128 15161 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:55.997748 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:55.932544 15173 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:08:55.931437 14869 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:55.992429 12791 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1535) took 2505 ms (client timeout 2607 ms). Trace:
I20260202 15:08:55.998915 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:56.000159 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:56.001052 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:56.001519 13362 raft_consensus.cc:2468] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:56.001865 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:56.002208 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 400) took 1126 ms. Trace:
I20260202 15:08:55.886041 15127 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:56.002327 13358 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:56.190033 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:56.003100 13328 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:56.216974 12723 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.006s sys 0.010s
I20260202 15:08:56.003767 13359 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260202 15:08:56.129434 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1628) took 1297 ms. Trace:
I20260202 15:08:55.998680 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:56.175140 12615 connection.cc:232] RPC call timeout handler was delayed by -1.0285130110000864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
I20260202 15:08:55.975131 14851 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260202 15:08:56.263775 13361 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.017s sys 0.019s
W20260202 15:08:56.274333 12763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.006s sys 0.009s
W20260202 15:08:56.274384 12764 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.011s sys 0.005s
W20260202 15:08:56.344079 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:56.350805 12765 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.008s sys 0.013s
W20260202 15:08:56.352149 12647 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.005s sys 0.015s
I20260202 15:08:56.395273 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 817) took 2076 ms. Trace:
I20260202 15:08:55.972368 15024 consensus_queue.cc:1059] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.526s
I20260202 15:08:55.974647 15113 consensus_queue.cc:1059] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.524s
W20260202 15:08:56.706844 12725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.002s sys 0.027s
W20260202 15:08:56.458544 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 455472us
I20260202 15:08:57.000197 13354 rpcz_store.cc:276] 0202 15:08:54.321585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.375895 (+ 54310us) service_pool.cc:225] Handling call
0202 15:08:54.376166 (+ 271us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:54.389477 (+ 13311us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:56.395042 (+2005565us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b5400610154. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:56.395219 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29205376}
W20260202 15:08:56.408381 12615 connection.cc:232] RPC call timeout handler was delayed by -1.0033130110000457s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
I20260202 15:08:57.001402 13328 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:57.001737 12615 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:56.155587 15173 raft_consensus.cc:697] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:57.002240 15127 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:57.000914 13358 consensus_meta.cc:287] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.604s user 0.004s sys 0.005s
I20260202 15:08:57.007699 13358 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:57.008565 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 684) took 1034 ms. Trace:
I20260202 15:08:57.009032 13358 rpcz_store.cc:276] 0202 15:08:55.973944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:56.001665 (+ 27721us) service_pool.cc:225] Handling call
0202 15:08:57.008492 (+1006827us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:56.766676 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1619) took 1851 ms. Trace:
I20260202 15:08:56.398175 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 387) took 1109 ms. Trace:
W20260202 15:08:56.546906 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:56.999789 12725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.002s user 0.004s sys 0.037s
I20260202 15:08:57.010850 12642 rpcz_store.cc:276] 0202 15:08:54.915293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.536380 (+621087us) service_pool.cc:225] Handling call
0202 15:08:55.536618 (+ 238us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:55.570398 (+ 33780us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:55.570432 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:55.570867 (+ 435us) log.cc:844] Serialized 26 byte log entry
0202 15:08:56.394723 (+823856us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:56.394755 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:56.394818 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:56.766458 (+371640us) raft_consensus.cc:1685] finished
0202 15:08:56.766482 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:56.766615 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33467,"raft.queue_time_us":37954,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":4763,"thread_start_us":485791,"threads_started":1,"wal-append.queue_time_us":1056170}
W20260202 15:08:56.999750 12763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.255s user 0.009s sys 0.025s
I20260202 15:08:57.011356 12720 rpcz_store.cc:276] 0202 15:08:55.288587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.405206 (+116619us) service_pool.cc:225] Handling call
0202 15:08:55.405496 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.491665 (+ 86169us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:56.397885 (+906220us) spinlock_profiling.cc:243] Waited 74.3 ms on lock 0x7b5400031e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:56.398104 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":170834176}
W20260202 15:08:57.013224 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 1.28435e+06us
W20260202 15:08:57.013492 12764 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.272s user 0.011s sys 0.023s
W20260202 15:08:57.013819 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 1.28222e+06us
W20260202 15:08:56.653483 13363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.008s sys 0.016s
W20260202 15:08:56.866758 13357 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.013s sys 0.009s
W20260202 15:08:56.398377 12723 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.705s user 0.011s sys 0.010s
W20260202 15:08:57.017097 12765 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.278s user 0.015s sys 0.023s
W20260202 15:08:57.018313 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 1.28732e+06us
I20260202 15:08:56.627494 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:08:57.015396 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:57.020184 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:57.021191 13358 raft_consensus.cc:2393] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:08:56.397083 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:56.397218 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1638) took 1205 ms. Trace:
W20260202 15:08:57.017107 13357 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.852s user 0.015s sys 0.010s
I20260202 15:08:56.401180 12645 rpcz_store.cc:276] 0202 15:08:54.832312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.170579 (+338267us) service_pool.cc:225] Handling call
0202 15:08:55.171914 (+ 1335us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.172080 (+ 166us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:08:55.481089 (+309009us) log.cc:844] Serialized 14 byte log entry
0202 15:08:56.129056 (+647967us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:56.129117 (+ 61us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:56.129172 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:56.129249 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:56.129375 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":134532,"raft.queue_time_us":160968,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":96,"spinlock_wait_cycles":31680896,"thread_start_us":729488,"threads_started":2,"wal-append.queue_time_us":735127}
I20260202 15:08:57.022125 12644 rpcz_store.cc:276] 0202 15:08:55.191300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.559911 (+368611us) service_pool.cc:225] Handling call
0202 15:08:55.560135 (+ 224us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.660759 (+100624us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:55.660787 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:55.660857 (+ 70us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:55.660914 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:56.396283 (+735369us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b54000127d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:56.396342 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:56.396484 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230966144}
I20260202 15:08:57.023044 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.002485 13354 raft_consensus.cc:1240] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
W20260202 15:08:57.017161 12647 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.201s user 0.010s sys 0.028s
I20260202 15:08:57.026099 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1628) took 1213 ms. Trace:
W20260202 15:08:57.013139 13361 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.494s user 0.028s sys 0.026s
W20260202 15:08:57.028643 13363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.093s user 0.011s sys 0.019s
W20260202 15:08:56.998859 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:57.033318 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:57.033859 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:57.034266 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:56.518949 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1632) took 1685 ms. Trace:
I20260202 15:08:57.041570 12649 rpcz_store.cc:276] 0202 15:08:54.833073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.411888 (+578815us) service_pool.cc:225] Handling call
0202 15:08:55.510302 (+ 98414us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.510573 (+ 271us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:08:55.510596 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:55.510638 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:55.510663 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:55.512607 (+ 1944us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:56.518744 (+1006137us) spinlock_profiling.cc:243] Waited 98.2 ms on lock 0x7b540020d850. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:56.518884 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2340,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":10609,"spinlock_wait_cycles":225781632}
I20260202 15:08:57.026561 12647 rpcz_store.cc:276] 0202 15:08:55.812983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.813124 (+ 141us) service_pool.cc:225] Handling call
0202 15:08:55.813383 (+ 259us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:55.813563 (+ 180us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:55.813596 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:55.814029 (+ 433us) log.cc:844] Serialized 26 byte log entry
0202 15:08:57.025647 (+1211618us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:57.025700 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.025785 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:57.025856 (+ 71us) raft_consensus.cc:1685] finished
0202 15:08:57.025874 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.026033 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35584,"thread_start_us":1209260,"threads_started":1,"wal-append.queue_time_us":852610}
W20260202 15:08:56.990618 15155 consensus_meta.cc:287] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.082s user 0.003s sys 0.007s
W20260202 15:08:55.999154 12791 rpcz_store.cc:269] 0202 15:08:53.486835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:53.487395 (+ 560us) service_pool.cc:225] Handling call
0202 15:08:55.992237 (+2504842us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b54003570d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:55.992364 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1689005824}
I20260202 15:08:57.043501 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:57.042224 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 685) took 1066 ms. Trace:
I20260202 15:08:57.024415 12645 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:57.047701 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.048524 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:57.049160 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:56.264257 15326 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.017s sys 0.040s
W20260202 15:08:56.627836 14681 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.822s user 0.000s sys 0.027s
I20260202 15:08:56.069006 15324 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.059309 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.061322 12641 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:57.071782 12649 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
W20260202 15:08:57.072074 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1630) took 2239 ms (client timeout 2245 ms). Trace:
I20260202 15:08:57.072297 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 444) took 1159 ms. Trace:
W20260202 15:08:57.072378 12641 rpcz_store.cc:269] 0202 15:08:54.832684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.047334 (+2214650us) service_pool.cc:225] Handling call
0202 15:08:57.072028 (+ 24694us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.072609 12649 rpcz_store.cc:276] 0202 15:08:55.912823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.059056 (+1146233us) service_pool.cc:225] Handling call
0202 15:08:57.072253 (+ 13197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.051909 13354 rpcz_store.cc:276] 0202 15:08:55.975788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.001884 (+1026096us) service_pool.cc:225] Handling call
0202 15:08:57.002305 (+ 421us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:57.041956 (+ 39651us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.042177 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:56.076315 12792 raft_consensus.cc:1240] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 7. Ops: []
W20260202 15:08:56.396520 15116 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.007s sys 0.014s
W20260202 15:08:56.397892 15179 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.009s sys 0.012s
I20260202 15:08:57.061631 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 432) took 1218 ms. Trace:
I20260202 15:08:57.078454 12647 rpcz_store.cc:276] 0202 15:08:55.842944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.043181 (+1200237us) service_pool.cc:225] Handling call
0202 15:08:57.061583 (+ 18402us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:56.621872 15225 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.047104 15341 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:57.064230 15326 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.495s user 0.028s sys 0.059s
W20260202 15:08:57.071296 14681 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.261s user 0.002s sys 0.041s
I20260202 15:08:56.176160 15033 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.067063 15155 raft_consensus.cc:2749] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:56.217216 15262 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.002s sys 0.015s
I20260202 15:08:56.156544 15356 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.181067 15356 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election won for term 7
W20260202 15:08:57.181246 15376 log.cc:927] Time spent T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20260202 15:08:57.181654 15356 raft_consensus.cc:697] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:56.500327 15260 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:56.125278 15117 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.061100 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 430) took 1218 ms. Trace:
I20260202 15:08:57.068991 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:57.212173 13359 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:56.071558 15297 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.213557 14681 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.213943 15297 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:56.164951 15352 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:56.989498 15373 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.180318 14869 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:56.062588 15019 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:56.668742 15095 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.015391 14851 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:57.076759 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:56.067512 15166 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.082780 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:57.226866 12647 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:57.064281 12642 raft_consensus.cc:2368] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7.
W20260202 15:08:57.227623 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 427) took 1386 ms (client timeout 1768 ms). Trace:
W20260202 15:08:57.228003 12642 rpcz_store.cc:269] 0202 15:08:55.841147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.048918 (+1207771us) service_pool.cc:225] Handling call
0202 15:08:57.227567 (+178649us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:57.070611 15035 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.007s sys 0.012s
W20260202 15:08:56.763509 15362 log.cc:927] Time spent T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.240s user 0.000s sys 0.000s
I20260202 15:08:57.182649 15326 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.071755 15127 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:57.077983 15116 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.282s user 0.010s sys 0.025s
W20260202 15:08:57.081809 15179 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.206s user 0.014s sys 0.023s
I20260202 15:08:57.163524 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:57.211979 12645 rpcz_store.cc:276] 0202 15:08:55.842435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.022755 (+1180320us) service_pool.cc:225] Handling call
0202 15:08:57.061030 (+ 38275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.238168 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.239123 12645 raft_consensus.cc:2410] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260202 15:08:57.182464 15262 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.486s user 0.002s sys 0.026s
I20260202 15:08:57.214171 14681 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.073354 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.226210 14851 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:57.214417 15297 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.232429 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:57.263950 12642 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:57.230129 15035 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.011s user 0.007s sys 0.012s
I20260202 15:08:57.130517 15225 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.178033 15341 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.083137 15355 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:57.239770 13354 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:57.251571 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 428) took 1410 ms. Trace:
I20260202 15:08:57.291298 12645 rpcz_store.cc:276] 0202 15:08:55.841416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.237935 (+1396519us) service_pool.cc:225] Handling call
0202 15:08:57.251519 (+ 13584us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.218092 15117 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.222340 15155 raft_consensus.cc:3055] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
W20260202 15:08:57.178011 12532 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 9fae829b1c574bc795c5a424036329b3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 66b8c715b9e6479e9218b08bd3156380 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:57.295419 12532 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 117456us
I20260202 15:08:57.182812 15033 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.267278 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 433) took 1415 ms. Trace:
I20260202 15:08:57.300098 12649 rpcz_store.cc:276] 0202 15:08:55.852255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.073164 (+1220909us) service_pool.cc:225] Handling call
0202 15:08:57.267187 (+194023us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.300997 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.301707 12649 raft_consensus.cc:2368] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8.
I20260202 15:08:57.302060 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 437) took 1416 ms. Trace:
I20260202 15:08:57.270043 12647 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
W20260202 15:08:57.272451 12726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.078s sys 0.145s
I20260202 15:08:57.238193 15378 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.292042 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:57.303849 12726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s user 0.079s sys 0.145s
I20260202 15:08:57.302281 12642 raft_consensus.cc:2410] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260202 15:08:57.304062 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 435) took 1442 ms. Trace:
I20260202 15:08:57.304343 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 426) took 1463 ms. Trace:
I20260202 15:08:57.304395 12645 rpcz_store.cc:276] 0202 15:08:55.861240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.291833 (+1430593us) service_pool.cc:225] Handling call
0202 15:08:57.304016 (+ 12183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.232507 15095 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.075399 15173 consensus_queue.cc:237] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.224452 15352 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.251885 15127 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.292843 12689 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:57.227299 15373 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.305356 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 386) took 2017 ms. Trace:
I20260202 15:08:57.299778 15155 raft_consensus.cc:740] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:57.306476 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1631) took 1385 ms. Trace:
I20260202 15:08:57.306912 13359 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:57.319159 12647 rpcz_store.cc:276] 0202 15:08:55.920621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.082508 (+1161887us) service_pool.cc:225] Handling call
0202 15:08:57.306417 (+223909us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.308142 12646 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:57.310506 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:57.320133 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.320403 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 431) took 1477 ms. Trace:
I20260202 15:08:57.320833 12645 rpcz_store.cc:276] 0202 15:08:55.842666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.310321 (+1467655us) service_pool.cc:225] Handling call
0202 15:08:57.320346 (+ 10025us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.321638 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.322247 12645 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:57.323278 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:57.230284 14869 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.313126 12644 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:57.325914 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 446) took 1412 ms. Trace:
I20260202 15:08:57.326329 12647 rpcz_store.cc:276] 0202 15:08:55.913407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.319890 (+1406483us) service_pool.cc:225] Handling call
0202 15:08:57.325853 (+ 5963us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.304678 12642 rpcz_store.cc:276] 0202 15:08:55.840865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.232225 (+1391360us) service_pool.cc:225] Handling call
0202 15:08:57.304288 (+ 72063us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.252377 15326 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:57.316591 12722 rpcz_store.cc:276] 0202 15:08:55.288346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.391553 (+103207us) service_pool.cc:225] Handling call
0202 15:08:55.391852 (+ 299us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:55.457952 (+ 66100us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:55.457982 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:55.458394 (+ 412us) log.cc:844] Serialized 26 byte log entry
0202 15:08:55.981937 (+523543us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:55.981987 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:55.982067 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:57.305185 (+1323118us) raft_consensus.cc:1685] finished
0202 15:08:57.305203 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.305309 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65711,"raft.queue_time_us":71945,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5407,"thread_start_us":495424,"threads_started":1,"wal-append.queue_time_us":815116}
I20260202 15:08:57.302393 12649 rpcz_store.cc:276] 0202 15:08:55.885663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.300730 (+1415067us) service_pool.cc:225] Handling call
0202 15:08:57.302010 (+ 1280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.329505 12649 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:57.327296 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.329944 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 434) took 1473 ms. Trace:
I20260202 15:08:57.330253 12649 rpcz_store.cc:276] 0202 15:08:55.856403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.329151 (+1472748us) service_pool.cc:225] Handling call
0202 15:08:57.329366 (+ 215us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:57.329780 (+ 414us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.329905 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.332140 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1629) took 1453 ms. Trace:
I20260202 15:08:57.332465 12649 rpcz_store.cc:276] 0202 15:08:55.879222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.331559 (+1452337us) service_pool.cc:225] Handling call
0202 15:08:57.331762 (+ 203us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:57.331898 (+ 136us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:57.331917 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:57.331948 (+ 31us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:57.331976 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.332016 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.332101 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.334280 12649 raft_consensus.cc:1240] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:57.334729 12645 raft_consensus.cc:2393] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 8: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:57.335893 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 441) took 1424 ms. Trace:
I20260202 15:08:57.336133 12645 rpcz_store.cc:276] 0202 15:08:55.911794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.335376 (+1423582us) service_pool.cc:225] Handling call
0202 15:08:57.335589 (+ 213us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:57.335723 (+ 134us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:57.335735 (+ 12us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:57.335757 (+ 22us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:57.335775 (+ 18us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.335802 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.335865 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.336861 12645 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: []
I20260202 15:08:57.332454 12644 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
W20260202 15:08:57.337395 12690 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.005s sys 0.037s
W20260202 15:08:57.337651 12690 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.720s user 0.005s sys 0.037s
W20260202 15:08:57.337872 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 720431us
I20260202 15:08:57.343883 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 826) took 1938 ms. Trace:
I20260202 15:08:57.344151 13363 rpcz_store.cc:276] 0202 15:08:55.405693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.408094 (+ 2401us) service_pool.cc:225] Handling call
0202 15:08:55.408401 (+ 307us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.408593 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:55.935132 (+526539us) log.cc:844] Serialized 14 byte log entry
0202 15:08:57.343696 (+1408564us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:57.343724 (+ 28us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:57.343746 (+ 22us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.343780 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.343840 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1298622,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"thread_start_us":1934420,"threads_started":3,"wal-append.queue_time_us":1393599}
I20260202 15:08:57.318079 15116 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:57.345631 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 439) took 1434 ms. Trace:
I20260202 15:08:57.299171 15355 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.291033 15356 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.348050 12649 rpcz_store.cc:276] 0202 15:08:55.911144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.333777 (+1422633us) service_pool.cc:225] Handling call
0202 15:08:57.334104 (+ 327us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:57.345372 (+ 11268us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.345577 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.348984 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 438) took 1438 ms. Trace:
I20260202 15:08:57.350575 12647 rpcz_store.cc:276] 0202 15:08:55.910768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.327017 (+1416249us) service_pool.cc:225] Handling call
0202 15:08:57.348948 (+ 21931us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:57.308880 15378 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:57.353658 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 352286us
I20260202 15:08:57.354799 13328 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:57.355268 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 330182us
W20260202 15:08:57.356230 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:57.362787 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:57.363241 12796 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:57.349906 15116 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.372291 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:57.372762 12799 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:57.361860 12764 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:57.364185 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 340448us
I20260202 15:08:57.372989 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:57.378755 12649 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:57.383572 12647 raft_consensus.cc:3055] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:57.383996 12647 raft_consensus.cc:740] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:57.384776 12647 consensus_queue.cc:260] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.385574 12647 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:57.385957 12796 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:57.362928 15166 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.362668 15372 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:57.362470 15179 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:57.374891 15116 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:57.409564 12649 raft_consensus.cc:2410] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20260202 15:08:57.441787 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 87066.4us
W20260202 15:08:57.443858 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 63552.5us
W20260202 15:08:57.440168 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 95019.6us
I20260202 15:08:57.400981 15113 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:57.445188 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 83406us
I20260202 15:08:57.453620 12799 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:08:57.454640 12800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 382) took 2897 ms (client timeout 2867 ms). Trace:
W20260202 15:08:57.431087 15384 log.cc:927] Time spent T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
W20260202 15:08:57.431277 15385 log.cc:927] Time spent T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
I20260202 15:08:57.489192 13326 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:57.455200 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:57.455639 12800 rpcz_store.cc:269] 0202 15:08:54.557424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.454458 (+2897034us) service_pool.cc:212] Skipping call since client already timed out
0202 15:08:57.454581 (+ 123us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260202 15:08:57.458760 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:57.439196 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 437461us
I20260202 15:08:57.448863 13328 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:57.449117 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:08:57.472911 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:57.413954 15033 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.490227 13359 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 7: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:08:57.510541 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:57.432561 15297 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:57.490613 13326 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:57.403371 14681 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.510484 12649 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
W20260202 15:08:57.545176 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:57.568827 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:57.432318 15019 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.451499 15341 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.574352 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 681) took 1616 ms. Trace:
I20260202 15:08:57.445681 15392 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.437633 15225 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:57.566918 15365 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.228s user 0.000s sys 0.001s
W20260202 15:08:57.577994 13362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 649) took 1743 ms (client timeout 1625 ms). Trace:
I20260202 15:08:57.511811 15117 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.581960 12649 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8.
I20260202 15:08:57.618963 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:57.560300 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:57.583328 13357 rpcz_store.cc:276] 0202 15:08:55.957611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.957791 (+ 180us) service_pool.cc:225] Handling call
0202 15:08:55.983659 (+ 25868us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.983885 (+ 226us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:57.022382 (+1038497us) log.cc:844] Serialized 14 byte log entry
0202 15:08:57.388721 (+366339us) log.cc:844] Serialized 14 byte log entry
0202 15:08:57.573667 (+184946us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:57.573708 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:57.573746 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.573817 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.573926 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25965,"prepare.queue_time_us":815307,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":71,"raft.queue_time_us":96534,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":11933,"spinlock_wait_cycles":27983616,"thread_start_us":1372638,"threads_started":3}
I20260202 15:08:57.378839 15155 consensus_queue.cc:260] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:57.490223 15382 log.cc:927] Time spent T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.185s user 0.000s sys 0.001s
I20260202 15:08:57.512634 13354 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
W20260202 15:08:57.592274 13362 rpcz_store.cc:269] 0202 15:08:55.834925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.835056 (+ 131us) service_pool.cc:225] Handling call
0202 15:08:57.577781 (+1742725us) spinlock_profiling.cc:243] Waited 44.5 ms on lock 0x7b540068ef54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:57.577916 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":102316288}
I20260202 15:08:57.402040 15166 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:57.593573 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 103043us
I20260202 15:08:57.577842 15397 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.584885 15297 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:57.620144 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.660610 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:57.632611 12689 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:57.574993 15033 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.661861 15033 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.412747 15372 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.414531 15179 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:57.641942 15391 log.cc:927] Time spent T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.262s user 0.000s sys 0.000s
I20260202 15:08:57.618739 15352 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.512753 15113 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.617461 15392 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:57.647423 15379 log.cc:927] Time spent T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.295s user 0.000s sys 0.000s
I20260202 15:08:57.617655 15225 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.648228 15420 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:57.651300 15408 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:57.685844 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 53301.6us
I20260202 15:08:57.633186 15117 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.587503 15095 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.655776 15166 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:57.658545 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1582) took 2111 ms. Trace:
I20260202 15:08:57.696893 12649 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:57.664913 14869 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.641700 15326 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:57.664165 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:57.668828 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 158101us
I20260202 15:08:57.437969 15394 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:57.591671 15127 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:57.675323 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 222411us
W20260202 15:08:57.683353 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 234546us
I20260202 15:08:57.656052 15155 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:57.686686 12799 raft_consensus.cc:1240] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 3. Current term is 4. Ops: []
I20260202 15:08:57.687624 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:57.660501 13326 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:57.664738 15324 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.664574 15411 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.666055 15019 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.670233 15397 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:57.905691 13356 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 637) took 2910 ms (client timeout 2782 ms). Trace:
I20260202 15:08:57.934892 15426 raft_consensus.cc:2749] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:57.774520 15422 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.814375 15116 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.682578 15297 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.253s
I20260202 15:08:57.729112 15095 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:57.935904 15422 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.936738 15095 raft_consensus.cc:697] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:08:57.936934 15389 log.cc:927] Time spent T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.619s user 0.000s sys 0.000s
I20260202 15:08:57.937330 15116 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.937769 15095 consensus_queue.cc:237] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.938126 15422 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:57.920009 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 259589us
I20260202 15:08:57.938189 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1563) took 2234 ms. Trace:
I20260202 15:08:57.939675 12794 rpcz_store.cc:276] 0202 15:08:55.703468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.723985 (+ 20517us) service_pool.cc:225] Handling call
0202 15:08:55.742848 (+ 18863us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:56.217595 (+474747us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:56.217637 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:56.396409 (+178772us) log.cc:844] Serialized 26 byte log entry
0202 15:08:57.071231 (+674822us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:57.071270 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.071331 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:57.936398 (+865067us) spinlock_profiling.cc:243] Waited 3.92 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:57.937917 (+ 1519us) raft_consensus.cc:1685] finished
0202 15:08:57.937938 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.938126 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1734,"raft.queue_time_us":80449,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"spinlock_wait_cycles":9026432,"thread_start_us":928509,"threads_started":2,"wal-append.queue_time_us":812867}
I20260202 15:08:57.941391 15253 consensus_queue.cc:1059] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:08:57.942122 15253 raft_consensus.cc:3055] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:57.942433 15253 raft_consensus.cc:740] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:57.739969 15421 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:57.943199 15253 consensus_queue.cc:260] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.944553 15253 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:57.697060 12723 rpcz_store.cc:276] 0202 15:08:55.547563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.587216 (+ 39653us) service_pool.cc:225] Handling call
0202 15:08:55.587529 (+ 313us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.587754 (+ 225us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:08:57.017928 (+1430174us) log.cc:844] Serialized 14 byte log entry
0202 15:08:57.640834 (+622906us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:57.640954 (+ 120us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:57.658287 (+ 17333us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.658362 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.658482 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1177108,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":256175,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":14681,"thread_start_us":1947125,"threads_started":3,"wal-append.queue_time_us":637051}
I20260202 15:08:57.946313 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 424) took 3099 ms. Trace:
I20260202 15:08:57.946691 12650 rpcz_store.cc:276] 0202 15:08:54.846424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.444734 (+598310us) service_pool.cc:225] Handling call
0202 15:08:55.445094 (+ 360us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.484794 (+ 39700us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.946088 (+2461294us) spinlock_profiling.cc:243] Waited 38.6 ms on lock 0x7b540020e254. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:57.946259 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88862464}
I20260202 15:08:57.947069 15375 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:08:57.947957 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:57.764274 15402 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.948531 15375 consensus_queue.cc:1048] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:08:57.949077 15402 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:56.397297 12797 rpcz_store.cc:276] 0202 15:08:54.875580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.973080 (+ 97500us) service_pool.cc:225] Handling call
0202 15:08:55.025957 (+ 52877us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:55.344936 (+318979us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:55.344966 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:55.364569 (+ 19603us) log.cc:844] Serialized 26 byte log entry
0202 15:08:55.850258 (+485689us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:55.850291 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:55.850348 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:56.001985 (+151637us) raft_consensus.cc:1685] finished
0202 15:08:56.002006 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:56.002153 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":52545,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":102,"thread_start_us":704948,"threads_started":2,"wal-append.queue_time_us":630056,"wal-append.run_cpu_time_us":1687,"wal-append.run_wall_time_us":1002039}
W20260202 15:08:57.950328 12795 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 403) took 3038 ms (client timeout 1638 ms). Trace:
I20260202 15:08:57.948611 12650 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:57.950688 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 395) took 2088 ms (client timeout 2632 ms). Trace:
I20260202 15:08:57.950984 15402 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.688369 15225 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.812249 15401 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:57.951146 12717 rpcz_store.cc:269] 0202 15:08:55.862492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.862605 (+ 113us) service_pool.cc:225] Handling call
0202 15:08:57.950518 (+2087913us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b54000177d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:57.950635 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44678912}
I20260202 15:08:57.951830 15401 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.952612 15225 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.953714 15401 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.808841 15410 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.954540 15410 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:57.950726 12795 rpcz_store.cc:269] 0202 15:08:54.911971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.920093 (+ 8122us) service_pool.cc:225] Handling call
0202 15:08:57.950130 (+3030037us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b5400371ed4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:57.950250 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1203311872}
I20260202 15:08:57.815886 15179 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.954070 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 381) took 3397 ms. Trace:
I20260202 15:08:57.740967 15318 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.729544 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 829) took 2219 ms. Trace:
I20260202 15:08:57.731200 14851 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.956610 13361 rpcz_store.cc:276] 0202 15:08:55.510043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.510128 (+ 85us) service_pool.cc:225] Handling call
0202 15:08:55.510352 (+ 224us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:55.510495 (+ 143us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:55.510514 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:55.518838 (+ 8324us) log.cc:844] Serialized 26 byte log entry
0202 15:08:57.028367 (+1509529us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:57.028434 (+ 67us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.028520 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:57.729348 (+700828us) raft_consensus.cc:1685] finished
0202 15:08:57.729373 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.729485 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1509155,"threads_started":1,"wal-append.queue_time_us":1612858}
I20260202 15:08:57.957405 12641 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 8: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:57.957937 12796 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:57.958228 14851 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.764535 15113 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.958933 12645 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: []
I20260202 15:08:57.943177 15421 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:57.958649 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1632) took 1895 ms (client timeout 1729 ms). Trace:
W20260202 15:08:57.959892 12641 rpcz_store.cc:269] 0202 15:08:56.063395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.076501 (+1013106us) service_pool.cc:225] Handling call
0202 15:08:57.958459 (+881958us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b540008acd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:57.958592 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1680523008}
I20260202 15:08:57.772382 15414 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.960665 15414 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.955895 12792 rpcz_store.cc:276] 0202 15:08:54.557126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.368990 (+811864us) service_pool.cc:225] Handling call
0202 15:08:56.035281 (+666291us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:57.078082 (+1042801us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.817391 (+739309us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b54003570d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:57.953889 (+136498us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b54003570d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:57.954016 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1550992512}
I20260202 15:08:57.688721 14681 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.811360 15372 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:57.962944 14681 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.807957 15400 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.963958 15400 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.964970 14681 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.965942 15400 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.696976 15408 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:57.861461 15419 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:08:57.899240 15347 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.729139 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:57.883234 15259 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:57.968575 15408 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:58.014623 12647 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.017s sys 0.000s
I20260202 15:08:57.968994 15411 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:57.759960 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:57.686633 15396 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:57.766479 15378 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:57.970290 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 445) took 2057 ms (client timeout 1723 ms). Trace:
I20260202 15:08:57.970185 15324 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:57.698917 15117 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:57.971407 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:57.739773 15392 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.698792 15390 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:08:57.972380 12615 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:57.972698 15373 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.687053 15352 consensus_queue.cc:1059] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.286s
I20260202 15:08:57.697438 15033 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.973968 15421 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:57.956279 15410 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:57.962587 15414 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:57.864168 15106 raft_consensus.cc:3055] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:57.976472 12649 raft_consensus.cc:1240] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:57.947875 12764 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:57.865978 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1627) took 2173 ms. Trace:
I20260202 15:08:57.869671 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:57.979269 13356 rpcz_store.cc:269] 0202 15:08:54.995254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.905478 (+2910224us) service_pool.cc:212] Skipping call since client already timed out
0202 15:08:57.905609 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260202 15:08:57.813158 15260 consensus_meta.cc:287] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.598s user 0.005s sys 0.004s
I20260202 15:08:57.816637 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1576) took 2327 ms. Trace:
I20260202 15:08:57.982009 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1622) took 3739 ms. Trace:
I20260202 15:08:57.699041 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 380) took 3142 ms. Trace:
W20260202 15:08:57.911859 12724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 388) took 2623 ms (client timeout 1612 ms). Trace:
I20260202 15:08:57.864585 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 394) took 2005 ms. Trace:
W20260202 15:08:57.699294 12718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1766) took 3469 ms (client timeout 1935 ms). Trace:
I20260202 15:08:57.954875 15397 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:57.988875 13326 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:57.725023 12690 leader_election.cc:400] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5.
I20260202 15:08:57.992048 13328 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:57.992161 12650 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:58.007122 12796 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
W20260202 15:08:58.009343 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:57.974529 13355 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:58.013053 12688 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.008s sys 0.007s
W20260202 15:08:57.868191 15364 log.cc:927] Time spent T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 1.487s user 0.008s sys 0.011s
W20260202 15:08:57.815291 15367 log.cc:927] Time spent T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 1.291s user 0.003s sys 0.021s
W20260202 15:08:57.741411 15383 log.cc:927] Time spent T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.343s user 0.000s sys 0.001s
W20260202 15:08:58.015064 12647 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s user 0.017s sys 0.000s
W20260202 15:08:58.420460 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 428479us
W20260202 15:08:58.420831 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 431994us
I20260202 15:08:58.421209 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1635) took 1038 ms. Trace:
I20260202 15:08:58.421620 13328 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:58.421877 13326 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:58.034576 12765 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.003s sys 0.015s
W20260202 15:08:58.422977 12765 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.966s user 0.009s sys 0.034s
I20260202 15:08:58.416483 12690 leader_election.cc:403] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260202 15:08:58.423324 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 967045us
W20260202 15:08:58.423666 12644 rpcz_store.cc:269] 0202 15:08:55.913131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.048251 (+1135120us) service_pool.cc:225] Handling call
0202 15:08:57.970144 (+921893us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54002a8d54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:57.970232 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":575153024}
I20260202 15:08:58.416882 12643 rpcz_store.cc:276] 0202 15:08:55.692391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.692602 (+ 211us) service_pool.cc:225] Handling call
0202 15:08:55.692884 (+ 282us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.812102 (+119218us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.865711 (+2053609us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400280854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:57.865910 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":273051392}
I20260202 15:08:58.429111 13360 raft_consensus.cc:2393] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:08:58.402810 12689 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.024s sys 0.062s
W20260202 15:08:58.429591 12689 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s user 0.025s sys 0.062s
W20260202 15:08:58.429931 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 608805us
W20260202 15:08:58.351130 15173 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.034s sys 0.042s
W20260202 15:08:58.419122 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.421654 12647 rpcz_store.cc:276] 0202 15:08:57.383171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.383282 (+ 111us) service_pool.cc:225] Handling call
0202 15:08:57.383485 (+ 203us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:58.420743 (+1037258us) raft_consensus.cc:1534] Early marking committed up to index 1
0202 15:08:58.420768 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:58.420902 (+ 134us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:08:58.420933 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:58.420990 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:58.421082 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4469,"raft.queue_time_us":373002,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":1110,"thread_start_us":1028363,"threads_started":1}
I20260202 15:08:58.436796 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:57.845717 15407 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:58.419716 15394 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:58.418974 12725 rpcz_store.cc:276] 0202 15:08:55.859145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.859464 (+ 319us) service_pool.cc:225] Handling call
0202 15:08:55.859760 (+ 296us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:55.921101 (+ 61341us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:55.921140 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:55.943053 (+ 21913us) log.cc:844] Serialized 26 byte log entry
0202 15:08:57.012185 (+1069132us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:57.012240 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.864211 (+851971us) spinlock_profiling.cc:243] Waited 60.9 ms on lock 0x7b540019ad54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:57.864259 (+ 48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:57.864309 (+ 50us) raft_consensus.cc:1685] finished
0202 15:08:57.864326 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.864524 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2124,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":18290,"spinlock_wait_cycles":140183808,"thread_start_us":1014533,"threads_started":1,"wal-append.queue_time_us":1198259}
I20260202 15:08:58.046324 14681 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:57.843410 15406 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:58.420169 15408 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:58.424081 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1646) took 1037 ms. Trace:
W20260202 15:08:58.411003 15356 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.024s sys 0.016s
W20260202 15:08:58.425796 12688 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.041s user 0.008s sys 0.016s
W20260202 15:08:58.426126 12724 rpcz_store.cc:269] 0202 15:08:55.288831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.386638 (+ 97807us) service_pool.cc:225] Handling call
0202 15:08:57.911668 (+2525030us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b540009f1d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:57.911789 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":833898752}
W20260202 15:08:58.425666 12718 rpcz_store.cc:269] 0202 15:08:54.218409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.218945 (+ 536us) service_pool.cc:225] Handling call
0202 15:08:57.688135 (+3469190us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b54003acfd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:57.688255 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1090402688}
I20260202 15:08:58.425890 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1649) took 1026 ms. Trace:
I20260202 15:08:58.453636 12650 rpcz_store.cc:276] 0202 15:08:57.399341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.947721 (+548380us) service_pool.cc:225] Handling call
0202 15:08:58.425838 (+478117us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:58.424564 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:58.028515 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.425379 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1564) took 1066 ms. Trace:
I20260202 15:08:57.742260 15420 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:58.103744 15253 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:58.417944 15106 raft_consensus.cc:740] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:58.424432 12726 rpcz_store.cc:276] 0202 15:08:55.489604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.540191 (+ 50587us) service_pool.cc:225] Handling call
0202 15:08:56.398091 (+857900us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:56.616649 (+218558us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:56.616674 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:56.657138 (+ 40464us) log.cc:844] Serialized 52 byte log entry
0202 15:08:57.304226 (+647088us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:08:57.304261 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.304331 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:57.735834 (+431503us) raft_consensus.cc:1685] finished
0202 15:08:57.735858 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.816344 (+ 80486us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b5400031e50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:57.816571 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":218184,"raft.queue_time_us":223325,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":41252,"spinlock_wait_cycles":1972159744,"thread_start_us":607973,"threads_started":1,"wal-append.queue_time_us":658809}
I20260202 15:08:58.432186 12765 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:58.429061 15066 raft_consensus.cc:3055] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
W20260202 15:08:58.454806 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 1.07017e+06us
I20260202 15:08:58.461372 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:08:58.463685 12688 connection.cc:232] RPC call timeout handler was delayed by -1.0415845229999832s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260202 15:08:58.432577 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 484738us
I20260202 15:08:57.768319 15355 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:57.736145 15417 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:58.417039 13355 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:58.434190 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 461846us
I20260202 15:08:58.484354 12615 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:58.485915 12615 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:58.450556 15024 raft_consensus.cc:3055] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:08:58.423228 13326 leader_election.cc:400] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Vote denied by peer bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) with higher term. Message: Invalid argument: T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8.
I20260202 15:08:58.437384 15352 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260202 15:08:58.490342 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.035291 15259 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.437391 13361 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:58.496701 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.439405 12717 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: []
I20260202 15:08:58.497716 12647 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:58.011063 15419 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:58.447887 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.447811 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.504621 12723 raft_consensus.cc:2368] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6.
I20260202 15:08:58.417802 12798 rpcz_store.cc:276] 0202 15:08:54.556776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.618792 (+ 62016us) service_pool.cc:225] Handling call
0202 15:08:54.619113 (+ 321us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:54.656653 (+ 37540us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:54.656683 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:54.662965 (+ 6282us) log.cc:844] Serialized 26 byte log entry
0202 15:08:54.809423 (+146458us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:54.809480 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.687897 (+2878417us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b54001f90d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:57.687942 (+ 45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:57.687992 (+ 50us) raft_consensus.cc:1685] finished
0202 15:08:57.688006 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.698982 (+ 10976us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":65,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":25045,"spinlock_wait_cycles":85569920,"thread_start_us":146075,"threads_started":1,"wal-append.queue_time_us":234280,"wal-append.run_cpu_time_us":1843,"wal-append.run_wall_time_us":1017408}
I20260202 15:08:58.420115 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 451) took 1384 ms. Trace:
I20260202 15:08:58.505745 12645 rpcz_store.cc:276] 0202 15:08:57.035926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.343314 (+307388us) service_pool.cc:225] Handling call
0202 15:08:57.958780 (+615466us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:57.959297 (+ 517us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:58.419927 (+460630us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x7b540008acd0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:58.420063 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1414916864}
I20260202 15:08:58.451160 12649 rpcz_store.cc:276] 0202 15:08:57.386635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.976040 (+589405us) service_pool.cc:225] Handling call
0202 15:08:57.976281 (+ 241us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:58.423888 (+447607us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:58.424037 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:08:58.424692 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260202 15:08:58.479892 15066 raft_consensus.cc:740] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:08:58.454597 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.452687 12800 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 4.
I20260202 15:08:58.428661 15390 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:58.424113 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 699638us
W20260202 15:08:58.487920 15423 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.016s sys 0.014s
I20260202 15:08:58.456988 12725 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260202 15:08:58.437484 15173 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s user 0.034s sys 0.045s
I20260202 15:08:58.433682 15225 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 5
I20260202 15:08:58.428413 15318 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:58.471004 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.471725 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.433734 15260 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:58.473548 12793 raft_consensus.cc:1275] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260202 15:08:58.525055 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:58.516330 15423 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.559s user 0.017s sys 0.014s
I20260202 15:08:58.446822 15426 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:58.446673 15033 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:08:58.434958 13360 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 638) took 3414 ms (client timeout 3310 ms). Trace:
I20260202 15:08:58.478936 15420 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.486546 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.487406 12615 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:58.489686 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 66560.7us
W20260202 15:08:58.491196 12763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.000s sys 0.026s
W20260202 15:08:58.533682 12763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s user 0.001s sys 0.026s
I20260202 15:08:58.519270 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:58.491694 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.015326 15347 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.537596 15347 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:58.488559 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.492662 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.488924 12795 raft_consensus.cc:1275] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260202 15:08:58.549280 13362 raft_consensus.cc:2393] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:58.489135 12799 raft_consensus.cc:1275] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260202 15:08:58.494809 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.498526 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:58.418614 15396 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:58.486996 15327 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.006s sys 0.011s
I20260202 15:08:58.500442 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.501051 12688 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:58.503556 13361 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:58.490804 15417 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.506300 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.448896 15019 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:58.426047 12648 rpcz_store.cc:276] 0202 15:08:54.242131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.293542 (+ 51411us) service_pool.cc:225] Handling call
0202 15:08:54.293793 (+ 251us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:08:54.901164 (+607371us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:54.901191 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:08:54.916531 (+ 15340us) log.cc:844] Serialized 52 byte log entry
0202 15:08:55.344709 (+428178us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:08:57.812453 (+2467744us) log.cc:844] Serialized 14 byte log entry
0202 15:08:57.981273 (+168820us) log.cc:844] Serialized 14 byte log entry
0202 15:08:57.981690 (+ 417us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:57.981766 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:57.981813 (+ 47us) raft_consensus.cc:1685] finished
0202 15:08:57.981829 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:57.981954 (+ 125us) inbound_call.cc:173] Queueing success response
0202 15:08:58.421619 (+439665us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b180020c380. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea4b12d3 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad 0000000000379254 0000000000379099 00007f24ea49469a
Metrics: {"mutex_wait_us":3004830,"prepare.queue_time_us":2499674,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":72,"raft.queue_time_us":586386,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":2988,"spinlock_wait_cycles":413799936,"thread_start_us":546156,"threads_started":2,"wal-append.queue_time_us":1205642,"wal-append.run_cpu_time_us":1327,"wal-append.run_wall_time_us":1055665}
I20260202 15:08:58.556442 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.512748 15106 consensus_queue.cc:260] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.454597 12791 rpcz_store.cc:276] 0202 15:08:57.359243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.359382 (+ 139us) service_pool.cc:225] Handling call
0202 15:08:57.374396 (+ 15014us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:57.685689 (+311293us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:57.685718 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:08:57.685755 (+ 37us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:57.685784 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:58.425202 (+739418us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b5400116f54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:58.425240 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:58.425332 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":715497856}
W20260202 15:08:58.457530 15356 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s user 0.025s sys 0.016s
I20260202 15:08:58.508337 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.485426 15378 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:58.479298 15408 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:58.482393 15414 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:58.509656 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.510192 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.567627 12645 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:58.510499 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:58.510828 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.568490 12723 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:58.568780 12649 raft_consensus.cc:2410] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
W20260202 15:08:58.497414 15302 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.037s sys 0.088s
I20260202 15:08:58.511366 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.512128 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.515275 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.570204 12643 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:58.516286 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.488886 14869 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:08:58.501477 15155 consensus_meta.cc:287] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.564s user 0.007s sys 0.000s
I20260202 15:08:58.570356 12725 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:58.574563 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:58.517973 12531 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 78ef01280730461580740297403fe090 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica bbf140958c3b4f339c512a9dc4be2dab is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:08:58.575278 12531 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 57358us
I20260202 15:08:58.495597 15400 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:58.495445 15257 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:58.520751 12797 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:58.488310 15024 raft_consensus.cc:740] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:58.496662 15402 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:58.507798 15409 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.012s sys 0.042s
I20260202 15:08:58.456290 14681 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:58.524331 13355 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260202 15:08:58.437191 15397 raft_consensus.cc:2804] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
W20260202 15:08:58.514185 15095 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.009s sys 0.018s
I20260202 15:08:58.507133 15419 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:08:58.482230 15253 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.517103 15390 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Advancing to term 9
W20260202 15:08:58.530058 13360 rpcz_store.cc:269] 0202 15:08:55.020587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.862654 (+2842067us) service_pool.cc:225] Handling call
0202 15:08:58.434879 (+572225us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:08:58.451875 15070 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.006s sys 0.014s
I20260202 15:08:58.456079 15407 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:58.534030 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 845694us
I20260202 15:08:58.487972 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.601035 12718 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:58.563731 15426 consensus_queue.cc:1059] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
W20260202 15:08:58.535166 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 47856.6us
I20260202 15:08:58.535323 13326 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:58.482786 15406 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:58.550421 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.552524 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.605907 13357 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:58.553069 12720 raft_consensus.cc:2410] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260202 15:08:58.553550 12647 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:58.551210 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.554775 12689 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:58.607357 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 52668.5us
I20260202 15:08:58.608532 12689 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:58.555301 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:58.610983 12726 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:58.531635 15432 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:58.557070 12648 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:58.479441 15373 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:58.615883 15432 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.616266 15373 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.617697 15432 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:58.557693 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 56734.8us
W20260202 15:08:58.618801 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 83528.7us
I20260202 15:08:58.619719 12688 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:58.620419 13326 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:58.620576 12688 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:58.561825 13354 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:58.622215 12724 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:08:58.553716 15327 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.644s user 0.006s sys 0.012s
I20260202 15:08:58.582842 12725 raft_consensus.cc:2410] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260202 15:08:58.571720 14869 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.583976 12723 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8.
I20260202 15:08:58.563968 15019 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
W20260202 15:08:58.573544 15302 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.760s user 0.038s sys 0.089s
I20260202 15:08:58.573907 15414 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.594434 13355 pending_rounds.cc:85] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Aborting all ops after (but not including) 7
I20260202 15:08:58.641319 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.642060 13355 pending_rounds.cc:107] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260202 15:08:58.642362 13355 raft_consensus.cc:2889] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:58.539448 15066 consensus_queue.cc:260] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.596057 12644 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:58.582638 15117 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:58.533025 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.604164 12615 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:58.594411 15409 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.922s user 0.012s sys 0.043s
I20260202 15:08:58.631170 15432 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260202 15:08:58.572579 15408 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.596487 15402 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.613683 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.615043 15406 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:58.615173 15253 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:58.622586 12689 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:58.622985 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.675815 13357 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:58.621879 13326 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:58.676393 15444 log.cc:927] Time spent T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260202 15:08:58.676657 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 54866.1us
W20260202 15:08:58.677533 15440 log.cc:927] Time spent T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260202 15:08:58.677990 15392 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:58.678558 15392 raft_consensus.cc:697] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:58.679512 15392 consensus_queue.cc:237] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:58.600201 15070 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s user 0.007s sys 0.014s
W20260202 15:08:58.635414 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1630) took 2720 ms (client timeout 1681 ms). Trace:
W20260202 15:08:58.683525 12646 rpcz_store.cc:269] 0202 15:08:55.908207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.998496 (+ 90289us) service_pool.cc:225] Handling call
0202 15:08:58.628164 (+2629668us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b540029a754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:58.628261 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2080367488}
I20260202 15:08:58.617993 15372 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:58.684363 15035 consensus_queue.cc:1059] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:08:58.684561 15372 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.685946 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1648) took 1298 ms. Trace:
I20260202 15:08:58.618427 15421 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:58.627760 15318 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260202 15:08:58.626133 15437 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:08:58.596122 15095 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s user 0.010s sys 0.018s
I20260202 15:08:58.672698 15409 consensus_queue.cc:1059] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.016s
I20260202 15:08:58.650353 12650 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
W20260202 15:08:58.651003 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.617961 15373 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:58.655980 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 51887us
I20260202 15:08:58.699168 12615 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:58.671310 12726 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:08:58.634123 15439 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:58.641285 15356 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:08:58.643024 15019 raft_consensus.cc:697] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:58.644232 15355 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:58.677345 14869 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:58.689810 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.690083 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:58.688877 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.690188 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.734638 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.747655 12650 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
I20260202 15:08:58.748873 13358 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:58.749913 12614 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:58.550949 15434 raft_consensus.cc:2749] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:08:58.694015 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:58.689155 13361 raft_consensus.cc:1240] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260202 15:08:58.690505 13357 raft_consensus.cc:1240] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260202 15:08:58.602481 15024 consensus_queue.cc:260] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.686296 12641 rpcz_store.cc:276] 0202 15:08:57.391748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:58.011866 (+620118us) service_pool.cc:225] Handling call
0202 15:08:58.012147 (+ 281us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:58.019844 (+ 7697us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:58.019889 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:58.020330 (+ 441us) log.cc:844] Serialized 26 byte log entry
0202 15:08:58.513668 (+493338us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:08:58.513725 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:58.513824 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:58.685729 (+171905us) raft_consensus.cc:1685] finished
0202 15:08:58.685748 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:58.685890 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":336,"raft.queue_time_us":22203,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":29838,"thread_start_us":492163,"threads_started":1,"wal-append.queue_time_us":537232}
I20260202 15:08:58.728734 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:58.716485 12615 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:58.688659 15409 raft_consensus.cc:3055] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:58.750013 13326 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:58.758826 12648 raft_consensus.cc:2393] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 7: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:08:58.759248 13328 connection.cc:232] RPC call timeout handler was delayed by -2.1044609279999804s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1612s for a response
W20260202 15:08:58.759569 13328 connection.cc:232] RPC call timeout handler was delayed by -2.043128877000072s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
I20260202 15:08:58.687486 15437 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:58.647899 15431 consensus_queue.cc:1059] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.085s
I20260202 15:08:58.672977 15259 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:58.686694 15372 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:58.753155 15439 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:58.789129 13326 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:58.758111 14851 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260202 15:08:58.745419 15438 log.cc:927] Time spent T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
W20260202 15:08:58.792589 13328 connection.cc:232] RPC call timeout handler was delayed by -1.1494467980000991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
I20260202 15:08:58.689888 15407 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:58.795018 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:58.704190 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 81661us
I20260202 15:08:58.801322 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.697034 15257 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260202 15:08:58.729869 15418 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:58.745052 15035 raft_consensus.cc:3055] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:08:58.730211 15411 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:58.727985 15421 raft_consensus.cc:697] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:58.687903 15408 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:58.717549 15420 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:58.820446 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 338710us
I20260202 15:08:58.728879 15066 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:58.749563 15400 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:08:58.753332 15155 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:58.746457 15396 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:58.824589 13328 connection.cc:232] RPC call timeout handler was delayed by -1.0270467980001285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260202 15:08:58.826098 13360 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:58.823294 14851 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:58.830250 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 209731us
I20260202 15:08:58.891702 12765 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:08:58.902846 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.820240 15439 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:58.831041 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 114643us
I20260202 15:08:58.703750 15117 consensus_queue.cc:1059] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.122s
I20260202 15:08:58.884755 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.753661 15355 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.885494 12689 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:58.915364 12725 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
W20260202 15:08:58.839339 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.855424 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.884445 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.832453 15035 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:58.884722 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.833909 15418 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:08:58.801941 15437 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:58.756939 15409 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:58.826360 15024 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:58.876659 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:58.935832 12717 raft_consensus.cc:2410] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260202 15:08:58.931114 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 330435us
I20260202 15:08:58.849025 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.826053 15417 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:58.899129 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.907114 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.954964 12615 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:58.907930 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.824296 15434 consensus_queue.cc:1059] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.103s
I20260202 15:08:58.908610 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:58.956494 12794 raft_consensus.cc:2368] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7.
I20260202 15:08:58.909250 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.910127 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:58.916582 13354 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:58.829679 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.958150 13361 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
W20260202 15:08:58.915558 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 48596.7us
I20260202 15:08:58.918720 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.918920 13358 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:58.919909 12796 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:08:58.921154 12688 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:58.916860 15400 raft_consensus.cc:2749] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:08:58.889214 15095 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:08:58.906970 14851 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:58.933183 15418 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:58.932564 15117 consensus_queue.cc:1243] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:08:58.937100 12763 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:58.907482 12765 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:58.888983 15408 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:58.918191 15259 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:58.971011 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.979146 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.979910 13361 raft_consensus.cc:2410] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260202 15:08:58.971612 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 50538.9us
I20260202 15:08:58.974135 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:58.980895 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 191801us
I20260202 15:08:58.910667 15447 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:58.894786 15155 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.909686 14681 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:58.979802 13358 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:58.920630 15378 raft_consensus.cc:697] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:58.981007 12688 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:58.976784 15440 log.cc:927] Time spent T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260202 15:08:58.982820 12614 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:58.983234 13326 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:58.953243 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:58.981493 13360 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:58.985046 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.954346 15410 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260202 15:08:58.985589 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 48567.9us
I20260202 15:08:59.002974 12763 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:59.003343 12764 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:59.004259 12764 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:08:59.004395 12763 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:58.986055 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 78690.1us
W20260202 15:08:58.992508 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.920840 15327 consensus_queue.cc:1059] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260202 15:08:58.995749 12688 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:58.983033 12648 pending_rounds.cc:85] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 6
I20260202 15:08:58.996896 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:58.912711 15019 consensus_queue.cc:237] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:58.973799 15397 raft_consensus.cc:697] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:58.960480 15414 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:58.921037 15390 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:58.974279 15095 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:59.062086 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:08:59.052546 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.028340 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 630) took 4783 ms. Trace:
I20260202 15:08:59.059705 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.011976 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:59.081811 12648 pending_rounds.cc:107] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 5.7
W20260202 15:08:59.021999 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 504408us
I20260202 15:08:59.031188 12646 raft_consensus.cc:3055] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:59.083492 12648 raft_consensus.cc:2889] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:59.083639 13358 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:59.083890 12646 raft_consensus.cc:740] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:59.084030 12690 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:59.064553 12689 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:59.006146 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:59.082541 12721 raft_consensus.cc:2410] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260202 15:08:59.033835 12800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 410) took 1348 ms (client timeout 1587 ms). Trace:
W20260202 15:08:59.086126 12800 rpcz_store.cc:269] 0202 15:08:57.685482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.685945 (+ 463us) service_pool.cc:225] Handling call
0202 15:08:59.033698 (+1347753us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b5400116554. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:59.033784 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1015177088}
W20260202 15:08:59.086501 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.043550 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1575) took 3554 ms. Trace:
I20260202 15:08:59.083216 13355 rpcz_store.cc:276] 0202 15:08:54.245398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:54.320657 (+ 75259us) service_pool.cc:225] Handling call
0202 15:08:54.322191 (+ 1534us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:08:55.371038 (+1048847us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b4c0023ad40. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24eb836e50 00007f24eb847ca2 00007f24eb80e055 00007f24eb83535d 00007f24eb834fc3 00007f24ed559e0b 00007f24ed55a935 00007f24ed555bb0 00007f24ed55745e 00007f24ed5617c9 00007f24ed562096
0202 15:08:57.974247 (+2603209us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b540052db20. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4dd09f 00007f24ed555be0 00007f24ed55745e 00007f24ed5617c9 00007f24ed562096 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2
0202 15:08:58.642743 (+668496us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:59.028066 (+385323us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b540052d354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:08:59.028269 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1343809536}
W20260202 15:08:59.090008 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 84454us
I20260202 15:08:59.030674 15253 raft_consensus.cc:2749] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:59.087109 12719 rpcz_store.cc:276] 0202 15:08:55.489362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:55.518182 (+ 28820us) service_pool.cc:225] Handling call
0202 15:08:55.551323 (+ 33141us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:55.551598 (+ 275us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:55.551620 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:55.551998 (+ 378us) log.cc:844] Serialized 26 byte log entry
0202 15:08:55.973451 (+421453us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:08:55.973501 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:55.973575 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:57.814758 (+1841183us) spinlock_profiling.cc:243] Waited 98.1 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:08:58.509227 (+694469us) raft_consensus.cc:1685] finished
0202 15:08:58.509270 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:59.043393 (+534123us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b540033bb50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:59.043495 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4109,"raft.run_cpu_time_us":55,"raft.run_wall_time_us":69,"spinlock_wait_cycles":301052160,"thread_start_us":421119,"threads_started":1,"wal-append.queue_time_us":798767}
I20260202 15:08:59.033504 15106 raft_consensus.cc:3060] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:08:59.094102 15327 raft_consensus.cc:3055] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:08:59.016173 15324 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:59.094939 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:59.095633 13358 raft_consensus.cc:2410] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 9 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260202 15:08:59.011535 12792 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:08:59.097162 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 101461us
I20260202 15:08:59.097779 12688 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:59.098546 12688 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:58.988449 15259 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:58.978686 15373 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:08:59.102030 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:59.089610 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 85296us
I20260202 15:08:59.098836 15324 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:58.979254 15402 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.094544 15327 raft_consensus.cc:740] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:58.992059 15173 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:08:59.118546 12688 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:59.035666 15410 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:58.995996 15400 raft_consensus.cc:3055] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:08:59.066808 15451 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:08:59.027880 15458 log.cc:927] Time spent T 4ed5f5e04d064d6e8d431aa2209cc141 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260202 15:08:59.124562 12795 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:58.995313 15433 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:08:59.152083 15400 raft_consensus.cc:740] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
W20260202 15:08:59.054972 15428 log.cc:927] Time spent T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.498s user 0.003s sys 0.000s
I20260202 15:08:59.034806 15117 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:59.128978 15355 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:59.056288 15297 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:59.031253 15426 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:59.006407 14681 raft_consensus.cc:2804] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:59.027222 15155 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:59.136543 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 49572.7us
I20260202 15:08:59.118913 15445 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:59.149382 15372 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:08:59.125064 15166 consensus_queue.cc:1048] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.047s
I20260202 15:08:59.023528 15378 consensus_queue.cc:237] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:08:59.150580 15449 log.cc:927] Time spent T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
I20260202 15:08:59.121706 15417 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:59.150776 15327 consensus_queue.cc:260] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:58.995846 15447 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.173398 15407 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:08:59.084550 12646 consensus_queue.cc:260] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.208766 12646 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.099978 15095 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Advancing to term 9
W20260202 15:08:59.152904 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 46876.4us
I20260202 15:08:59.005601 15409 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.121877 15456 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:59.114405 15373 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:59.117970 15324 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:59.165019 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 46555.1us
I20260202 15:08:59.165720 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:08:59.167286 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 108596us
W20260202 15:08:59.119493 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 207245us
I20260202 15:08:59.168555 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:58.985605 15421 consensus_queue.cc:237] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.025897 15396 consensus_queue.cc:1059] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.134s
I20260202 15:08:59.158265 15451 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:59.165323 15394 consensus_meta.cc:287] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.710s user 0.004s sys 0.006s
I20260202 15:08:59.158787 14869 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:08:58.996057 15408 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:59.183003 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.101861 15419 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:59.185948 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 49783.7us
I20260202 15:08:58.987990 15035 consensus_queue.cc:260] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.163352 15117 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:08:59.187860 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 72463.1us
I20260202 15:08:59.130043 15414 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:59.165403 15173 raft_consensus.cc:697] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:59.171257 15297 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:59.265990 15297 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:59.197283 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 242385us
I20260202 15:08:59.269847 12615 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:59.161325 15341 consensus_queue.cc:1048] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.027s
I20260202 15:08:59.151759 15410 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:59.271291 15410 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:59.165449 15423 consensus_queue.cc:1048] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
W20260202 15:08:59.275151 15462 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260202 15:08:59.182294 14681 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:59.275699 14681 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:59.163637 15432 consensus_queue.cc:1048] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
W20260202 15:08:59.276556 15460 log.cc:927] Time spent T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260202 15:08:59.113876 15259 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.281565 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.281896 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:08:59.210426 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.282276 12800 raft_consensus.cc:1217] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Deduplicated request from leader. Original: 3.6->[5.7-6.8] Dedup: 6.8->[]
I20260202 15:08:59.190575 15155 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:59.282737 12800 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: [5.7-6.8]
I20260202 15:08:59.282999 15155 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.283959 15396 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:59.284410 15396 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.285132 15155 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.191020 15433 raft_consensus.cc:697] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:59.285326 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.286010 12797 raft_consensus.cc:2368] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6.
I20260202 15:08:59.286283 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.286005 15396 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.226588 12644 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260202 15:08:59.286864 12793 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:59.229120 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:59.287089 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:59.287487 13361 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:59.287765 12794 raft_consensus.cc:2393] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:08:59.225052 13359 raft_consensus.cc:2393] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 8: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:08:59.289338 15390 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:59.292265 12792 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:08:59.293123 15439 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.295600 15447 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.199021 15400 consensus_queue.cc:260] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 5, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.296468 15400 raft_consensus.cc:3060] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:59.296422 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:08:59.297055 15225 consensus_meta.cc:287] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.774s user 0.004s sys 0.004s
I20260202 15:08:59.297331 12792 raft_consensus.cc:2410] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:59.297408 15225 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5.
I20260202 15:08:59.210119 15417 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.300387 15417 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:59.301640 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.304531 15408 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:59.306353 15420 raft_consensus.cc:3055] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:08:59.306828 15420 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:59.307940 15420 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.309417 15420 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:59.231433 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:59.310889 12765 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:59.311239 13356 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:08:59.311655 15396 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:08:59.311956 15396 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:59.312007 12765 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.312245 15327 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:59.312371 15396 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:59.312714 15430 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:59.314030 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 425) took 3473 ms. Trace:
I20260202 15:08:59.255997 15394 raft_consensus.cc:2749] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:08:59.314340 12642 rpcz_store.cc:276] 0202 15:08:55.840550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:57.327501 (+1486951us) service_pool.cc:225] Handling call
0202 15:08:58.628358 (+1300857us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:08:58.635050 (+ 6692us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:08:58.635088 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:08:58.643517 (+ 8429us) log.cc:844] Serialized 26 byte log entry
0202 15:08:58.935897 (+292380us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:08:58.935943 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:08:58.936001 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:08:59.300777 (+364776us) raft_consensus.cc:1685] finished
0202 15:08:59.300796 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:08:59.313815 (+ 13019us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540029a750. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:59.313978 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":513,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":10054,"spinlock_wait_cycles":2990834560,"thread_start_us":256957,"threads_started":1,"wal-append.queue_time_us":539801}
I20260202 15:08:59.249079 15451 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.264739 14869 raft_consensus.cc:697] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:08:59.265164 15117 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:59.319114 15430 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:59.321280 15409 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:59.225364 12798 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:08:59.234146 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.210556 15347 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:59.335664 15173 consensus_queue.cc:237] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.336504 15347 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:59.336875 15347 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:59.236625 12646 raft_consensus.cc:1275] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260202 15:08:59.208312 15407 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:08:59.241184 15418 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.209426 15372 raft_consensus.cc:697] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:08:59.252272 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.216153 15456 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:59.255004 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.372790 12650 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:08:59.255672 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.219751 15373 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.192265 15397 consensus_queue.cc:237] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.230345 15324 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.256021 12690 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:08:59.379043 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 123089us
I20260202 15:08:59.380270 15423 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260202 15:08:59.380302 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.380694 15423 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:59.381042 15423 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:59.258679 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.382735 12690 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:59.286347 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.287648 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.306890 15419 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.315382 13363 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:08:59.318599 14869 consensus_queue.cc:237] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:08:59.229794 12764 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:59.384625 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 154890us
I20260202 15:08:59.230593 12688 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:59.386972 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 163392us
I20260202 15:08:59.387428 13361 raft_consensus.cc:1275] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260202 15:08:59.233541 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.235253 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.388577 12647 raft_consensus.cc:2393] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 7: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
W20260202 15:08:59.225780 13328 leader_election.cc:336] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Timed out: RequestConsensusVote RPC to 127.11.46.194:44367 timed out after 1.612s (SENT)
I20260202 15:08:59.389079 12645 raft_consensus.cc:2393] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:59.389261 13328 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:59.252841 15033 consensus_meta.cc:287] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.721s user 0.008s sys 0.002s
I20260202 15:08:59.390043 15033 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:08:59.390110 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 164341us
W20260202 15:08:59.323962 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:59.390856 13328 leader_election.cc:336] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.638s (SENT)
W20260202 15:08:59.391582 13328 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.723s (SENT) [suppressed 14 similar messages]
I20260202 15:08:59.323294 12649 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:08:59.392181 15419 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:08:59.324415 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 54636.6us
I20260202 15:08:59.392657 15419 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:59.393054 15419 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.393966 12615 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.394340 15259 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.395004 15033 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:08:59.395967 15259 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.398372 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:59.328230 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.403666 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.387902 15352 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:08:59.404107 15457 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260202 15:08:59.404559 12725 raft_consensus.cc:2410] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260202 15:08:59.329305 12717 raft_consensus.cc:1240] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260202 15:08:59.331727 15463 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:59.333679 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.407228 12723 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:59.339902 12646 pending_rounds.cc:85] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 5
I20260202 15:08:59.408022 12646 pending_rounds.cc:107] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260202 15:08:59.408363 12646 raft_consensus.cc:2889] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:59.352556 12722 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:08:59.357051 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.359114 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:08:59.376483 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.409942 12644 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
I20260202 15:08:59.410713 12800 raft_consensus.cc:2410] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260202 15:08:59.387701 13359 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:59.324533 12689 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:08:59.411931 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 87460us
I20260202 15:08:59.412194 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:59.412663 12689 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:59.412668 13359 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:08:59.413522 15463 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:59.413487 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:59.399950 15446 raft_consensus.cc:2749] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260202 15:08:59.400408 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.439595 13356 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260202 15:08:59.439954 13356 pending_rounds.cc:85] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Aborting all ops after (but not including) 6
I20260202 15:08:59.440155 13328 leader_election.cc:400] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6.
I20260202 15:08:59.440258 13356 pending_rounds.cc:107] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.7
I20260202 15:08:59.440570 13328 leader_election.cc:403] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260202 15:08:59.440783 13356 raft_consensus.cc:2917] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Aborting config change with OpId 6.7: Aborted: Op aborted by new leader
I20260202 15:08:59.441084 15446 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.441612 13356 raft_consensus.cc:2889] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 8 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:08:59.447379 15408 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:08:59.448069 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.400519 15324 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.387496 12688 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:08:59.258193 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.427321 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:59.419576 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:59.469789 12723 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:08:59.455796 12615 connection.cc:232] RPC call timeout handler was delayed by -2.3923022369999671s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1754s for a response
I20260202 15:08:59.466976 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:59.265044 15414 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:08:59.470763 12615 connection.cc:232] RPC call timeout handler was delayed by -2.3491022370001247s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18020000000000003s for a response
W20260202 15:08:59.471184 12615 connection.cc:232] RPC call timeout handler was delayed by -2.045802237000089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21389999999999998s for a response
W20260202 15:08:59.471590 12615 connection.cc:232] RPC call timeout handler was delayed by -1.8307022369999686s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2378s for a response
I20260202 15:08:59.472193 12615 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:08:59.474561 12615 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.476938 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.318476 15396 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.478510 15225 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260202 15:08:59.484354 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.485782 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:08:59.478958 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 91540.3us
I20260202 15:08:59.486711 12615 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.479720 15155 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:59.449842 14851 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.482626 15347 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.376327 15439 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:59.354101 15407 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:59.482739 15453 raft_consensus.cc:3055] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:59.483448 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:59.410362 15423 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.509919 13361 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.414077 15463 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:08:59.481455 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 77627.8us
I20260202 15:08:59.496850 15113 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:08:59.376376 15456 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:08:59.476788 15070 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260202 15:08:59.533280 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 401) took 1073 ms. Trace:
I20260202 15:08:59.533703 12722 rpcz_store.cc:276] 0202 15:08:58.459555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:58.461176 (+ 1621us) service_pool.cc:225] Handling call
0202 15:08:59.533098 (+1071922us) spinlock_profiling.cc:243] Waited 855 ms on lock 0x7b54000181d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:08:59.533215 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1966792448}
I20260202 15:08:59.535051 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:59.535899 12722 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 8: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:59.536895 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.537038 12615 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.538137 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.482995 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:59.469854 15302 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:08:59.478849 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.541498 12644 raft_consensus.cc:2410] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260202 15:08:59.490849 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.542349 12646 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:59.484143 15033 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:08:59.477547 15414 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.545099 15347 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:59.456193 15318 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.004s sys 0.009s
I20260202 15:08:59.555418 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:59.523643 15434 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:59.492163 15225 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:59.498517 15155 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.494192 15173 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:08:59.530591 12723 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:59.470012 15445 consensus_queue.cc:1048] T 965bcb05a43147b7be53449d0fe46509 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260202 15:08:59.492672 15473 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260202 15:08:59.539371 13326 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:08:59.541355 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 113131us
I20260202 15:08:59.622306 12690 leader_election.cc:400] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7.
W20260202 15:08:59.505059 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 56089.4us
I20260202 15:08:59.543206 12722 raft_consensus.cc:2393] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:59.449985 15437 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.551580 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:59.627024 12796 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:59.614367 15341 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:08:59.556145 12793 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:59.556815 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.631825 12797 raft_consensus.cc:2393] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:59.557442 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:59.523156 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.633167 12643 raft_consensus.cc:2410] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:08:59.558709 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.467343 15352 consensus_queue.cc:1059] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
W20260202 15:08:59.559333 15318 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.005s sys 0.009s
I20260202 15:08:59.564172 15225 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:59.574522 12764 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.575284 15489 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:08:59.577273 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.580542 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:59.647482 12796 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8.
I20260202 15:08:59.581377 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.648535 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.648953 12646 raft_consensus.cc:2393] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 8: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:59.582180 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:08:59.649318 12796 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:08:59.650115 12644 raft_consensus.cc:2393] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:08:59.525254 15113 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:08:59.654855 15113 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:08:59.586066 15372 consensus_queue.cc:237] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.587273 15434 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.659322 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.660487 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.661195 15437 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.583211 13361 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:08:59.585175 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:59.661790 15434 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.663118 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.663775 12794 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:59.664577 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.540607 15302 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:59.665573 15302 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:08:59.286058 15433 consensus_queue.cc:237] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.589644 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:59.667424 12723 raft_consensus.cc:2368] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 7. Current term is 8.
I20260202 15:08:59.589713 12615 leader_election.cc:304] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:08:59.668094 12721 raft_consensus.cc:1240] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:59.661051 12791 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
W20260202 15:08:59.668972 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 94511.8us
I20260202 15:08:59.668256 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:08:59.669783 12764 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.625526 13328 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:59.670936 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.586731 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:59.509716 15453 raft_consensus.cc:740] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:59.686604 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:59.687197 12792 raft_consensus.cc:2393] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:08:59.639729 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.636793 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.642166 12649 pending_rounds.cc:85] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 6
I20260202 15:08:59.653275 15423 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.556474 15400 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260202 15:08:59.672807 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 83171.5us
I20260202 15:08:59.673463 12723 raft_consensus.cc:1240] T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7. Ops: []
I20260202 15:08:59.614269 15173 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.674431 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:08:59.733032 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:59.733673 13359 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:08:59.676323 12793 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
W20260202 15:08:59.680817 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 55382.4us
I20260202 15:08:59.681172 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:59.688798 12645 raft_consensus.cc:1275] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260202 15:08:59.689157 12649 pending_rounds.cc:107] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.7
I20260202 15:08:59.687215 13358 raft_consensus.cc:2393] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 7: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:08:59.663041 15471 log.cc:927] Time spent T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20260202 15:08:59.706549 12722 raft_consensus.cc:1240] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:59.689134 12797 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:08:59.707814 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:08:59.641408 15446 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6.
I20260202 15:08:59.689766 15066 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:08:59.745672 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.763751 12649 raft_consensus.cc:2917] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Aborting config change with OpId 6.7: Aborted: Op aborted by new leader
I20260202 15:08:59.721145 15396 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.693370 15253 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:08:59.742483 13355 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:08:59.689965 15024 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:59.715356 15431 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:08:59.716403 15324 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:59.709192 15418 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.800433 13328 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:08:59.777799 12649 raft_consensus.cc:2889] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260202 15:08:59.815776 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 146082us
I20260202 15:08:59.832479 13354 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:08:59.838497 13359 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
W20260202 15:08:59.826817 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:59.778609 15066 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:08:59.759709 14851 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:59.790426 15095 consensus_meta.cc:287] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.564s user 0.007s sys 0.000s
I20260202 15:08:59.847790 13358 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:08:59.719754 15410 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:59.833060 15409 consensus_meta.cc:287] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.000s
I20260202 15:08:59.860432 13328 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:08:59.861122 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:08:59.861828 13328 leader_election.cc:400] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6.
I20260202 15:08:59.862876 13328 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.822402 15431 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:08:59.867281 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.849726 15066 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.869184 12797 raft_consensus.cc:1240] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 4. Current term is 5. Ops: []
I20260202 15:08:59.840889 15024 raft_consensus.cc:697] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:08:59.869973 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.870826 12795 raft_consensus.cc:1240] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 3. Current term is 5. Ops: []
I20260202 15:08:59.849473 15420 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:08:59.874562 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:59.874626 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:08:59.864367 13328 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:08:59.865118 15117 consensus_meta.cc:287] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.546s user 0.008s sys 0.000s
I20260202 15:08:59.876549 12791 raft_consensus.cc:2376] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6. Re-sending same reply.
I20260202 15:08:59.878765 13328 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.880601 12791 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:08:59.865974 15445 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:08:59.868445 15260 raft_consensus.cc:2764] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260202 15:08:59.790489 15453 consensus_queue.cc:260] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.869858 15431 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.873407 15155 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:59.856101 15095 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.897215 15420 raft_consensus.cc:697] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:08:59.898069 15420 consensus_queue.cc:237] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.880029 15409 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election won for term 6
W20260202 15:08:59.893142 15451 consensus_meta.cc:287] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.577s user 0.008s sys 0.001s
I20260202 15:08:59.909802 15409 raft_consensus.cc:697] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:08:59.910389 15492 log.cc:927] Time spent T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260202 15:08:59.912060 13357 raft_consensus.cc:1240] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 8. Current term is 9. Ops: []
I20260202 15:08:59.902143 15453 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.897235 15446 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:08:59.884076 15117 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:59.908067 15419 consensus_meta.cc:287] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.514s user 0.011s sys 0.000s
I20260202 15:08:59.919494 15419 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.920732 15117 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.888165 15499 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260202 15:08:59.925443 15173 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:08:59.911424 15493 log.cc:927] Time spent T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260202 15:08:59.942209 15095 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:08:59.945083 15297 consensus_meta.cc:287] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.679s user 0.002s sys 0.006s
I20260202 15:08:59.945657 15297 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:08:59.951921 15454 raft_consensus.cc:3055] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:08:59.952502 15454 raft_consensus.cc:740] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:08:59.952960 15035 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:08:59.928862 15225 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.929947 15410 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:08:59.921128 15419 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:08:59.915174 15480 log.cc:927] Time spent T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20260202 15:08:59.911698 15451 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.953732 15454 consensus_queue.cc:260] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.904095 15373 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.910616 15409 consensus_queue.cc:237] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:08:59.975749 15481 log.cc:927] Time spent T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.338s user 0.002s sys 0.010s
W20260202 15:08:59.969777 15327 consensus_meta.cc:287] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.657s user 0.007s sys 0.002s
I20260202 15:08:59.975777 15373 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:59.986292 15451 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:08:59.982322 15456 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.985430 15454 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:08:59.994452 15456 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:08:59.991608 15463 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:08:59.997766 15463 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:09:00.030717 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:00.031998 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:08:58.516536 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:09:00.040769 12614 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:00.044694 12642 raft_consensus.cc:2410] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260202 15:09:00.046957 12641 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:09:00.047139 12545 catalog_manager.cc:5654] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:09:00.050917 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.038553 12615 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:08:59.898229 13328 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:00.156530 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 258200us
I20260202 15:09:00.157707 13328 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:00.066799 12720 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:00.069658 15302 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.160283 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1593) took 1667 ms. Trace:
I20260202 15:09:00.160835 15302 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.162771 15392 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260202 15:09:00.163189 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 538594us
I20260202 15:09:00.163623 15392 raft_consensus.cc:2749] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:00.160717 12720 rpcz_store.cc:276] 0202 15:08:58.492824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:58.606912 (+114088us) service_pool.cc:225] Handling call
0202 15:08:58.616030 (+ 9118us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:00.158832 (+1542802us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:00.159959 (+ 1127us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400146254. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:00.160199 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3334758784}
I20260202 15:09:00.070575 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:00.073424 15414 consensus_meta.cc:287] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.528s user 0.010s sys 0.000s
I20260202 15:08:59.981477 15431 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.078428 12764 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:00.084278 12792 raft_consensus.cc:1275] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:09:00.085640 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.087136 15070 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:00.087272 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:09:00.088052 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 47744.3us
I20260202 15:09:00.099298 15463 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
W20260202 15:09:00.100653 13363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 840) took 1653 ms (client timeout 1744 ms). Trace:
W20260202 15:09:00.103988 15407 consensus_meta.cc:287] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.586s user 0.011s sys 0.003s
I20260202 15:09:00.105340 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:08:59.925179 15024 consensus_queue.cc:237] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.121665 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.123032 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.124348 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.124409 12545 catalog_manager.cc:5654] T fe02793fdaa448caa739180083258aae P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 2 to 3, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:09:00.125706 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.188401 12791 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:00.189651 12690 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:00.127084 15297 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.191190 12545 catalog_manager.cc:5654] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 2 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260202 15:09:00.130539 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 92499.3us
W20260202 15:09:00.138931 15512 log.cc:927] Time spent T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.137s user 0.000s sys 0.002s
I20260202 15:09:00.192662 15302 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:09:00.143204 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:00.193183 15302 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:00.193527 13355 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:00.193750 15302 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:09:00.194330 12615 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:00.197073 15155 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:09:00.197667 15155 raft_consensus.cc:697] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:00.144435 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:00.198881 15155 consensus_queue.cc:237] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.199460 13362 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:09:00.151865 15432 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:00.152909 15225 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.153976 15113 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:00.201695 15432 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:00.202126 15432 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:00.173313 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:00.174224 15414 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:00.205304 15414 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.206410 12645 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 7: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:00.202489 15113 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.175252 15431 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.176158 13354 raft_consensus.cc:1240] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:00.127244 15173 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:00.207605 12545 catalog_manager.cc:5654] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 2, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 2 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:09:00.212083 13355 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:00.210980 12615 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:00.216967 12800 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:00.217335 12795 raft_consensus.cc:2393] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:09:00.230178 12798 consensus_meta.cc:287] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.598s user 0.008s sys 0.004s
I20260202 15:09:00.230584 15392 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260202 15:09:00.232698 14681 consensus_meta.cc:287] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.886s user 0.003s sys 0.011s
I20260202 15:09:00.233361 14681 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.234171 12717 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
W20260202 15:09:00.181011 13363 rpcz_store.cc:269] 0202 15:08:58.447190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:58.452368 (+ 5178us) service_pool.cc:225] Handling call
0202 15:09:00.100158 (+1647790us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54005d2d54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:00.100405 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1800046464}
I20260202 15:09:00.240154 14681 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.240630 15407 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.242894 15407 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.246976 15070 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:00.226119 15066 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.210630 12690 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:00.230588 12798 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:09:00.329599 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:09:00.340499 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 93495.5us
I20260202 15:09:00.341835 12797 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:09:00.342677 12794 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:09:00.347271 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:00.347149 12725 raft_consensus.cc:2393] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 8: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:00.348047 12791 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:09:00.350764 12800 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:00.348927 12614 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:00.347146 12545 catalog_manager.cc:5654] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260202 15:09:00.347126 15113 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:00.333424 15463 raft_consensus.cc:697] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:00.360487 12720 raft_consensus.cc:1240] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:00.363612 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:00.363885 12645 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:00.357435 15302 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.338009 15407 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:00.334586 15414 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:00.368470 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 290446us
I20260202 15:09:00.361200 15113 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:00.371737 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:00.372076 15456 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:00.373992 15453 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260202 15:09:00.374845 12544 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45928 (request call id 318) took 19926 ms (client timeout 15000 ms). Trace:
I20260202 15:09:00.375181 15070 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.375646 12724 raft_consensus.cc:1240] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:00.375712 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:09:00.376534 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 168264us
I20260202 15:09:00.386107 12794 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:00.387192 12722 raft_consensus.cc:1240] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:00.387754 12717 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:00.390873 15456 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.391158 15394 consensus_queue.cc:1059] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:09:00.392248 15394 raft_consensus.cc:3055] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:09:00.392582 15394 raft_consensus.cc:740] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:00.392901 15456 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.393425 15019 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:00.393903 15019 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:00.393489 15394 consensus_queue.cc:260] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.394361 15019 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:00.395262 15397 consensus_queue.cc:1059] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260202 15:09:00.396668 15397 consensus_queue.cc:1059] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:09:00.397648 13355 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:00.398064 15378 raft_consensus.cc:3055] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
W20260202 15:09:00.390450 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 182565us
I20260202 15:09:00.393986 15414 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.405596 12721 raft_consensus.cc:1240] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:00.406610 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:00.408214 12763 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.398217 15066 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.409929 12763 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.410351 15489 raft_consensus.cc:3055] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:00.410467 15066 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:09:00.410748 15489 raft_consensus.cc:740] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:00.363441 15297 consensus_queue.cc:1059] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260202 15:09:00.383118 15113 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:00.400941 12545 catalog_manager.cc:5654] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:09:00.417068 12545 catalog_manager.cc:5654] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:09:00.419503 12717 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:09:00.420260 15414 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.420945 12721 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:00.421576 15463 consensus_queue.cc:237] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.423103 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.423795 12646 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 3.
I20260202 15:09:00.424642 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.425107 12647 raft_consensus.cc:1240] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:00.425868 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.430958 12721 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 9.
W20260202 15:09:00.402534 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 167387us
I20260202 15:09:00.439916 13355 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:00.440101 12645 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 9.
I20260202 15:09:00.443312 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:09:00.378844 12544 rpcz_store.cc:269] 0202 15:08:40.433178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:40.433344 (+ 166us) service_pool.cc:225] Handling call
0202 15:08:42.139333 (+1705989us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b540023f854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c90c0 00007f24ed567bba 00007f24f32e8eac 00007f24f33e310e 00007f24f18de4a4 00007f24f18de432 00007f24f18de361 00007f24f18de2dc 00007f24f18dd592 00007f24eb89e34c 00007f24eb89d786
0202 15:09:00.355588 (+18216255us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:08:52.398333 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:08:52.635475 (+237142us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:08:52.635529 (+ 54us) write_op.cc:435] Acquired schema lock
0202 15:08:52.635540 (+ 11us) tablet.cc:660] Decoding operations
0202 15:08:52.835782 (+200242us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:08:52.940164 (+104382us) write_op.cc:641] Partition lock acquired for write op
0202 15:08:52.943499 (+ 3335us) tablet.cc:683] Acquiring locks for 7 operations
0202 15:08:53.196545 (+253046us) tablet.cc:699] Row locks acquired
0202 15:08:53.201516 (+ 4971us) write_op.cc:260] PREPARE: finished
0202 15:08:53.250481 (+ 48965us) write_op.cc:270] Start()
0202 15:08:53.310159 (+ 59678us) write_op.cc:276] Timestamp: P: 1770044933210261 usec, L: 0
0202 15:08:53.316504 (+ 6345us) op_driver.cc:348] REPLICATION: starting
0202 15:08:53.481559 (+165055us) log.cc:844] Serialized 2850 byte log entry
0202 15:08:54.636755 (+1155196us) op_driver.cc:464] REPLICATION: finished
0202 15:08:55.312527 (+675772us) write_op.cc:301] APPLY: starting
0202 15:08:55.312626 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0202 15:08:55.312843 (+ 217us) tablet.cc:1368] finished BulkCheckPresence
0202 15:08:55.327876 (+ 15033us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:08:55.328990 (+ 1114us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:08:55.387069 (+ 58079us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0202 15:08:55.459127 (+ 72058us) write_op.cc:312] APPLY: finished
0202 15:08:55.490742 (+ 31615us) log.cc:844] Serialized 59 byte log entry
0202 15:08:55.517596 (+ 26854us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:55.517734 (+ 138us) write_op.cc:454] Released schema lock
0202 15:08:55.517804 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0202 15:08:55.518231 (+ 427us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:08:55.518280 (+ 49us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2785036,"reported_tablets":25,"spinlock_wait_cycles":523164544,"child_traces":[["op",{"apply.queue_time_us":675437,"apply.run_cpu_time_us":3961,"apply.run_wall_time_us":205684,"num_ops":7,"prepare.queue_time_us":543044,"prepare.run_cpu_time_us":43152,"prepare.run_wall_time_us":1998564,"raft.queue_time_us":257496,"raft.run_cpu_time_us":93277,"raft.run_wall_time_us":481159,"replication_time_us":1318950,"thread_start_us":1775450,"threads_started":4,"wal-append.queue_time_us":733145,"wal-append.run_cpu_time_us":13821,"wal-append.run_wall_time_us":2444137}]]}
I20260202 15:09:00.444124 12723 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:00.445062 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:00.445188 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.400945 15378 raft_consensus.cc:740] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:00.456760 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.461951 12645 raft_consensus.cc:1240] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:00.459525 12615 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:00.464994 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:00.418591 12545 catalog_manager.cc:5654] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 2 to 5, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 5 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:09:00.396605 15502 consensus_queue.cc:1059] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260202 15:09:00.445063 15410 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:00.428928 15407 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:00.409981 15454 consensus_queue.cc:1048] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.056s
I20260202 15:09:00.461994 15070 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:00.475765 12648 pending_rounds.cc:85] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 2
I20260202 15:09:00.465385 15302 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.472859 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.480928 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1799) took 1044 ms. Trace:
I20260202 15:09:00.479317 12648 pending_rounds.cc:107] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260202 15:09:00.485586 12717 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 9: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:00.454991 15432 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.483706 12648 raft_consensus.cc:2889] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:00.483394 12725 rpcz_store.cc:276] 0202 15:08:59.436129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:59.476711 (+ 40582us) service_pool.cc:225] Handling call
0202 15:09:00.470548 (+993837us) spinlock_profiling.cc:243] Waited 780 ms on lock 0x7b540019ad54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:00.480877 (+ 10329us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1793645824}
I20260202 15:09:00.490622 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:09:00.490854 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 63469.6us
I20260202 15:09:00.491098 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.491691 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.493964 15352 consensus_queue.cc:1059] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:09:00.494091 12545 catalog_manager.cc:5654] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260202 15:09:00.481974 15117 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:00.481230 15070 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:09:00.501366 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 91569.4us
W20260202 15:09:00.502918 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 76097.5us
I20260202 15:09:00.504585 12615 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:00.505052 15378 consensus_queue.cc:260] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:00.506577 15378 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:00.506735 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:00.507767 13326 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:00.508738 12650 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:00.494995 15494 raft_consensus.cc:3055] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:00.510983 15494 raft_consensus.cc:740] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:00.503006 12722 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:09:00.511921 15494 consensus_queue.cc:260] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.513173 15494 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:00.514976 15431 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:00.515640 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.515480 15431 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.515978 12726 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:00.516185 12646 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:00.516752 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.517091 15318 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:00.517303 15431 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.517760 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.517539 15318 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:00.518240 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 79611.1us
I20260202 15:09:00.519035 15372 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:00.519719 15372 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.518322 12642 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:00.521730 15372 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.522167 15390 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:09:00.522856 15502 raft_consensus.cc:3055] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:00.523259 15502 raft_consensus.cc:740] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:00.501154 15394 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:00.524113 15502 consensus_queue.cc:260] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:00.525422 15502 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
W20260202 15:09:00.526405 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 47721.2us
I20260202 15:09:00.529632 12724 raft_consensus.cc:1275] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260202 15:09:00.531311 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.531728 12763 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.532366 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.532464 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.533171 12650 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:00.533406 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.533766 12763 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.533951 12649 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:00.533983 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.534505 15446 raft_consensus.cc:2804] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader election won for term 9
I20260202 15:09:00.535117 15446 raft_consensus.cc:697] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [term 9 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:00.535789 15454 raft_consensus.cc:2749] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:09:00.535917 15407 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.536513 15454 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:09:00.537977 15454 consensus_queue.cc:1059] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:09:00.502422 15489 consensus_queue.cc:260] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:00.540355 15489 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:00.544286 15503 raft_consensus.cc:3055] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:00.544667 15503 raft_consensus.cc:740] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:00.545331 12764 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:00.545475 15503 consensus_queue.cc:260] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.546361 12764 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:00.546972 15410 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:09:00.558858 15410 raft_consensus.cc:697] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:00.558934 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:00.559711 13361 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:00.559955 15019 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.500643 15070 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:09:00.529310 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 52806us
I20260202 15:09:00.561659 15378 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:00.565313 15378 raft_consensus.cc:697] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:00.566301 15378 consensus_queue.cc:237] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.569666 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:00.569952 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.559887 15410 consensus_queue.cc:237] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.556133 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:00.570688 12791 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:00.571548 15113 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:00.556743 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 52327us
I20260202 15:09:00.567760 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.568691 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.569340 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.560691 15432 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.571585 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:00.596262 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.596364 15019 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.596800 12642 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:00.597509 12646 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:09:00.605999 12723 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:00.607636 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.608136 12798 raft_consensus.cc:2410] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260202 15:09:00.608294 12642 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 9
I20260202 15:09:00.612071 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.612386 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:00.612581 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.613000 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.570461 15070 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.619055 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.620932 12646 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:09:00.622224 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.622367 12615 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:00.623255 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.622323 12797 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:00.626652 12800 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:00.626995 13326 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:00.609951 15503 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:00.634863 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:00.638612 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.639230 12646 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:00.644767 12726 raft_consensus.cc:1240] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:00.646381 12688 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:00.646797 12643 raft_consensus.cc:1240] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260202 15:09:00.647051 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.647450 15297 raft_consensus.cc:3055] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:00.647927 12795 raft_consensus.cc:2410] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260202 15:09:00.648399 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.649015 12643 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:00.648917 12688 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.649770 15106 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:00.650149 15106 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:00.650470 15106 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:00.651443 15318 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.536046 15446 consensus_queue.cc:237] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.654753 15392 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:00.655251 15392 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.655608 15070 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.657032 15392 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.647835 15297 raft_consensus.cc:740] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:00.599012 15454 consensus_queue.cc:1048] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260202 15:09:00.659417 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:00.634020 15494 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260202 15:09:00.669085 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 62719.8us
I20260202 15:09:00.686995 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:00.687595 13357 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:00.687708 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.687896 12615 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:00.688421 12798 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:00.689440 12615 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:00.690268 15070 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:00.690814 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.691282 13361 raft_consensus.cc:1240] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:00.663957 15019 raft_consensus.cc:2804] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:09:00.695066 15019 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:00.695431 15019 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:09:00.696938 15070 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:00.699016 15456 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:09:00.670395 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.676403 15024 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:00.669816 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.634727 15113 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:00.682179 15117 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:00.683154 12689 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:00.682682 15297 consensus_queue.cc:260] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:00.702061 12793 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:09:00.708423 12723 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:00.710708 12641 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:09:00.733078 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 43755.5us
W20260202 15:09:00.734226 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 200562us
I20260202 15:09:00.733134 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.713966 12688 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.738991 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.740904 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.743292 12646 raft_consensus.cc:2410] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260202 15:09:00.743901 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.745401 12615 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:00.707638 15070 raft_consensus.cc:2749] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:09:00.708979 15024 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:00.745607 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:00.756276 15411 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:00.758221 15522 log.cc:927] Time spent T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
W20260202 15:09:00.788436 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 105494us
I20260202 15:09:00.699828 15407 consensus_queue.cc:1059] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.097s
I20260202 15:09:00.760241 12763 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.761173 15173 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:00.762774 15456 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260202 15:09:00.763105 15463 consensus_queue.cc:1059] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260202 15:09:00.752735 12642 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 9.
I20260202 15:09:00.762466 12615 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:00.785316 12641 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:00.785980 12795 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:00.758939 15297 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:00.795781 15003 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:00.789446 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:00.759061 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 99867.4us
I20260202 15:09:00.731600 15432 consensus_queue.cc:1059] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.057s
I20260202 15:09:00.790340 12643 raft_consensus.cc:2410] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:09:00.791949 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.792407 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.797343 12642 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:00.800104 15494 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:00.791576 15024 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:00.788276 15392 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:00.802008 12793 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:00.805931 12615 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:00.806416 15463 consensus_queue.cc:1243] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260202 15:09:00.806942 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.809492 12646 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260202 15:09:00.813362 12724 raft_consensus.cc:2368] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7.
W20260202 15:09:00.815184 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 55051.8us
I20260202 15:09:00.824072 13357 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:00.825657 15392 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.826421 15463 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:00.800789 15113 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:00.834048 12614 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:00.792976 15106 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.831848 12793 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:00.796988 15155 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:09:00.800515 15411 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.847173 12642 raft_consensus.cc:1275] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260202 15:09:00.797235 15173 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:00.848577 12643 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:00.849773 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.859699 12763 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.867177 15433 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:00.868420 15117 raft_consensus.cc:3055] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Stepping down as leader of term 8
I20260202 15:09:00.869114 15302 raft_consensus.cc:3055] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:00.875142 12649 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:09:00.879914 12795 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:00.837175 15003 raft_consensus.cc:697] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:00.880717 12793 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:09:00.886595 12798 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:00.841611 15463 raft_consensus.cc:697] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:00.870865 15113 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:00.879017 15117 raft_consensus.cc:740] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:00.879523 15155 raft_consensus.cc:697] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:00.879451 15302 raft_consensus.cc:740] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:00.879344 14681 consensus_queue.cc:1059] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.038s
I20260202 15:09:00.927114 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:09:00.929061 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 69463.9us
I20260202 15:09:00.910879 15502 raft_consensus.cc:2749] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:00.914841 15113 raft_consensus.cc:697] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:00.916677 15394 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:00.953123 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:00.923666 15410 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:00.908214 15433 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.955410 12650 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
W20260202 15:09:00.961946 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 59650.7us
I20260202 15:09:00.959132 12763 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.969050 13326 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:00.972757 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:09:00.965909 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 62935.8us
I20260202 15:09:00.976089 12793 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260202 15:09:00.986726 15407 raft_consensus.cc:3055] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 8 LEADER]: Stepping down as leader of term 8
I20260202 15:09:00.987087 15407 raft_consensus.cc:740] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:00.988034 15407 consensus_queue.cc:260] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.989467 15407 raft_consensus.cc:3060] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:00.990283 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.991447 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:00.991914 12795 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:00.992205 15433 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:00.993636 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:00.994230 12724 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:09:00.995064 12690 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:00.995820 12765 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:00.996631 14681 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:00.996811 12690 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.997237 15463 consensus_queue.cc:237] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:00.997954 12690 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.977417 15003 consensus_queue.cc:237] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.003728 15456 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:09:01.004144 15225 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:01.005679 12690 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:00.969653 15392 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:00.982193 15297 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:09:01.006927 12688 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:01.010210 12798 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:09:01.011008 15035 consensus_queue.cc:1059] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20260202 15:09:01.012013 15260 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:01.012658 15390 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:01.013994 12795 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:01.021641 14681 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:09:00.985702 12689 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:01.031499 15524 log.cc:927] Time spent T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20260202 15:09:01.046715 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:01.085474 12795 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260202 15:09:01.086652 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 137143us
I20260202 15:09:01.088819 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:01.056737 15502 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260202 15:09:01.053141 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 47540.6us
W20260202 15:09:01.053836 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 46950.7us
I20260202 15:09:00.978382 15106 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:01.060108 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:01.046613 12797 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:01.062327 15414 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:01.062875 15117 consensus_queue.cc:260] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.069842 15113 consensus_queue.cc:237] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.071395 15411 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:01.072332 15173 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:01.090498 12798 raft_consensus.cc:1275] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260202 15:09:00.976821 15410 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.056197 15390 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:01.062189 15035 consensus_queue.cc:1243] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:09:01.067590 12764 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:00.985191 15394 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.074079 15432 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:09:01.106485 12688 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:09:01.110036 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 124419us
I20260202 15:09:01.110977 12721 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:01.097095 15225 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:09:01.050398 12799 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:09:01.126682 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 59219.6us
I20260202 15:09:01.102561 15414 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:01.128005 12764 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:01.054800 15302 consensus_queue.cc:260] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:01.122987 12765 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:01.132303 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:01.133870 12688 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:01.114935 15390 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Advancing to term 9
W20260202 15:09:01.150669 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 52805.9us
I20260202 15:09:01.084394 15260 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.162272 12725 raft_consensus.cc:1275] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260202 15:09:01.129309 15414 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:01.169523 13357 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:01.133145 14681 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:01.136943 15117 raft_consensus.cc:3060] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:01.196375 13361 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260202 15:09:01.199654 13361 pending_rounds.cc:85] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Aborting all ops after (but not including) 5
I20260202 15:09:01.130231 15297 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260202 15:09:01.201843 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 395993us
W20260202 15:09:01.202009 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 50627.2us
I20260202 15:09:01.205962 15394 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:09:01.170058 15508 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.032s sys 0.048s
I20260202 15:09:01.256155 12645 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260202 15:09:01.207593 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:01.210026 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:01.211395 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:01.212288 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:01.215574 13328 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:01.216001 15446 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:09:01.216791 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:01.260787 12690 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.217382 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:01.261796 13356 raft_consensus.cc:2393] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 9: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:01.217939 15417 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:09:01.221536 13357 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 9.
I20260202 15:09:01.080837 15155 consensus_queue.cc:237] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:01.151929 15489 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260202 15:09:01.227929 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 54812.2us
I20260202 15:09:01.195399 15432 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:01.199944 13361 pending_rounds.cc:107] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.6
I20260202 15:09:01.271131 13361 raft_consensus.cc:2917] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Aborting config change with OpId 4.6: Aborted: Op aborted by new leader
I20260202 15:09:01.258417 13360 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
W20260202 15:09:01.260579 15508 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.709s user 0.032s sys 0.049s
W20260202 15:09:01.251941 12791 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.087s sys 0.167s
I20260202 15:09:01.267812 15394 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 8 LEARNER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:01.223711 15411 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
W20260202 15:09:01.273833 12791 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.586s user 0.087s sys 0.167s
I20260202 15:09:01.275205 13362 raft_consensus.cc:2393] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 4: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:01.250027 15302 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:01.276609 13361 raft_consensus.cc:2889] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:01.250521 14681 raft_consensus.cc:697] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:01.251941 15070 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:09:01.281895 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 66411.3us
I20260202 15:09:01.268383 15489 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:01.270938 15035 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:01.285871 13328 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.273468 15417 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:01.293797 12615 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:01.296360 12763 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:01.338511 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 44812.5us
I20260202 15:09:01.265211 15225 consensus_queue.cc:1059] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260202 15:09:01.292992 15489 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:01.339112 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:01.283789 15297 consensus_queue.cc:1048] T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
W20260202 15:09:01.367653 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:01.337158 15019 consensus_meta.cc:287] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.611s user 0.011s sys 0.000s
I20260202 15:09:01.347519 15410 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:01.349906 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 65854us
W20260202 15:09:01.350143 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 61825.3us
I20260202 15:09:01.351223 15392 raft_consensus.cc:3055] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:01.351599 13359 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:09:01.339816 15407 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:01.352993 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:01.354034 15433 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:01.355230 14681 consensus_queue.cc:237] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:01.356149 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 74675.1us
I20260202 15:09:01.299366 15035 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.359949 12721 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:01.363269 15502 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:09:01.343359 15024 consensus_meta.cc:287] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.510s user 0.009s sys 0.001s
I20260202 15:09:01.352082 15542 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:01.372421 13328 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.299737 15394 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.069s
I20260202 15:09:01.345841 15378 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:01.295773 15070 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:01.389480 12763 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.391494 12690 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.300951 15417 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.374962 15392 raft_consensus.cc:740] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:01.362591 15489 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.371467 15019 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:01.389931 15502 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:01.386978 15173 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:01.387974 15433 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:01.466403 12717 raft_consensus.cc:1240] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:09:01.392942 15410 consensus_queue.cc:1059] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260202 15:09:01.457139 15463 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:01.472080 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 92451.8us
I20260202 15:09:01.433490 15407 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.456861 15260 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:01.455632 15024 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:01.475808 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 103463us
I20260202 15:09:01.463760 15378 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:01.493203 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 357640us
I20260202 15:09:01.476212 15540 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:01.482414 13328 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:01.498598 12725 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:01.501401 15546 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:01.469897 15173 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:01.501564 15539 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.214s user 0.002s sys 0.000s
W20260202 15:09:01.501698 15503 consensus_meta.cc:287] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.862s user 0.011s sys 0.000s
W20260202 15:09:01.525774 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 276726us
I20260202 15:09:01.508085 15003 raft_consensus.cc:2749] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:01.529244 12763 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.529896 12688 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:09:01.530031 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 47680.6us
I20260202 15:09:01.536772 12794 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:09:01.478384 15463 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.500291 15378 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.531944 15446 raft_consensus.cc:2804] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader election won for term 9
I20260202 15:09:01.502562 15540 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.492457 15392 consensus_queue.cc:260] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.576232 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:01.576860 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:01.578642 12723 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:01.569530 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:01.540346 15432 raft_consensus.cc:697] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:01.575623 12688 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:01.582366 12722 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:09:01.544934 15503 raft_consensus.cc:3055] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:01.564138 15394 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:01.585347 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:01.598327 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:01.510324 15546 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:01.604653 12724 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:01.609995 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:01.620014 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:01.586091 15503 raft_consensus.cc:740] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:09:01.621665 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 127423us
I20260202 15:09:01.622371 12719 raft_consensus.cc:2393] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:01.604416 15106 raft_consensus.cc:2749] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:01.605882 15070 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:01.630545 12722 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:09:01.624930 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:01.610267 15019 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:09:01.644169 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 365458us
I20260202 15:09:01.620573 15225 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:01.657810 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:01.623248 15392 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:01.664356 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:01.656533 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 265080us
I20260202 15:09:01.665918 12690 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:01.664952 12723 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:01.670003 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:01.622993 15417 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:01.629626 15542 raft_consensus.cc:697] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:01.669070 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:01.675087 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:01.678808 15559 consensus_queue.cc:1059] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260202 15:09:01.646801 15433 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:01.683284 13361 raft_consensus.cc:1275] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:09:01.655512 15407 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:01.687584 12794 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1585) took 1116 ms. Trace:
I20260202 15:09:01.680414 15035 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:01.687961 12794 rpcz_store.cc:276] 0202 15:09:00.567068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:00.569122 (+ 2054us) service_pool.cc:225] Handling call
0202 15:09:01.683578 (+1114456us) spinlock_profiling.cc:243] Waited 961 ms on lock 0x7b54002ebb54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:01.683677 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2210847104}
W20260202 15:09:01.697515 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:01.668709 15550 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:01.702169 12615 leader_election.cc:400] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 7. Current term is 8.
I20260202 15:09:01.702461 12615 leader_election.cc:403] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260202 15:09:01.674489 15225 raft_consensus.cc:697] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:09:01.712728 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 46909.3us
I20260202 15:09:01.700934 15559 consensus_queue.cc:1243] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer 6921687ed3814cccb2bd9c606875b9f0 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260202 15:09:01.707360 15544 log.cc:927] Time spent T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
I20260202 15:09:01.700393 15417 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:09:01.680718 15019 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
W20260202 15:09:01.720986 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:01.727172 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:01.727244 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:01.734014 13360 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:09:01.663224 15113 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.050s
I20260202 15:09:01.740097 12690 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.740774 12797 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:09:01.740895 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:01.747180 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:09:01.747349 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 78337.4us
I20260202 15:09:01.710187 15555 raft_consensus.cc:2804] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election won for term 5
W20260202 15:09:01.720806 15547 log.cc:927] Time spent T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260202 15:09:01.749627 12794 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:01.747823 12799 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260202 15:09:01.756561 12793 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:09:01.757525 12717 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 9.
I20260202 15:09:01.719123 15117 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 7. Current term is 8.
I20260202 15:09:01.759081 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:01.761596 15568 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:09:01.759673 15411 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:01.762142 15554 log.cc:927] Time spent T 00000000000000000000000000000000 P ec70d329e653431585e347713be6ee46: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20260202 15:09:01.762508 12690 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.762364 15542 consensus_queue.cc:237] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:01.763002 15551 log.cc:927] Time spent T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20260202 15:09:01.763185 15556 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader election won for term 9
I20260202 15:09:01.764478 15407 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:01.765215 15417 raft_consensus.cc:1064] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
I20260202 15:09:01.765935 15433 raft_consensus.cc:1064] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: attempting to promote NON_VOTER 66b8c715b9e6479e9218b08bd3156380 to VOTER
W20260202 15:09:01.767208 15561 log.cc:927] Time spent T cb4cc1899a25452c9c57398fe4022645 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260202 15:09:01.768352 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1640) took 1086 ms. Trace:
I20260202 15:09:01.773437 12720 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:01.776925 12615 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:01.723444 15446 raft_consensus.cc:697] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [term 9 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:01.741595 15559 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:01.782765 12794 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:01.742022 15024 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:01.788728 12798 raft_consensus.cc:2393] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:01.754153 15173 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:01.761981 15560 log.cc:927] Time spent T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260202 15:09:01.822862 15559 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.823426 15446 consensus_queue.cc:237] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.824787 15559 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:01.799170 15540 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:01.815289 12726 rpcz_store.cc:276] 0202 15:09:00.681633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:00.699670 (+ 18037us) service_pool.cc:225] Handling call
0202 15:09:00.735249 (+ 35579us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:00.782940 (+ 47691us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:00.782966 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:00.794190 (+ 11224us) log.cc:844] Serialized 52 byte log entry
0202 15:09:01.133163 (+338973us) raft_consensus.cc:1645] Marking committed up to 1
0202 15:09:01.133359 (+ 196us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:01.133548 (+ 189us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:01.768136 (+634588us) raft_consensus.cc:1685] finished
0202 15:09:01.768160 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:01.768296 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16248,"raft.queue_time_us":44500,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":5035,"thread_start_us":303848,"threads_started":1,"wal-append.queue_time_us":642640}
I20260202 15:09:01.788285 12765 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:09:01.827353 15562 log.cc:927] Time spent T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260202 15:09:01.827736 12765 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:01.789708 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:01.829342 13361 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:09:01.830319 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:01.830785 15584 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:01.831137 15454 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:01.831524 15454 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:01.831249 15584 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.701960 15432 consensus_queue.cc:237] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:01.784729 15410 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:01.776155 15378 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:01.798380 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:01.842283 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:01.786127 15555 raft_consensus.cc:697] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:09:01.797554 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 189053us
I20260202 15:09:01.801568 13355 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260202 15:09:01.795897 15494 consensus_queue.cc:1059] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260202 15:09:01.803768 15260 raft_consensus.cc:3055] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:01.804450 13362 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:01.804589 15546 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:01.805750 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:01.806843 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:01.800726 15302 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:01.807246 15557 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:01.807307 13357 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:09:01.714838 15550 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:01.819361 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 216293us
I20260202 15:09:01.830380 15003 raft_consensus.cc:3055] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:01.759905 15463 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:01.855970 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:01.797145 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1595) took 1178 ms. Trace:
W20260202 15:09:01.836333 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 307140us
I20260202 15:09:01.831875 15454 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:01.813766 15225 consensus_queue.cc:237] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:01.840576 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 80284.1us
I20260202 15:09:01.840633 15173 raft_consensus.cc:2749] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:01.794837 15035 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:01.842474 13359 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:01.802058 12690 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:01.820596 15556 raft_consensus.cc:697] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:01.820911 15407 raft_consensus.cc:697] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:01.859838 13362 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEARNER]: Advancing to term 7
I20260202 15:09:01.848733 13356 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:01.860611 12689 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.796161 15411 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:09:01.848618 15260 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:09:01.868530 15553 log.cc:927] Time spent T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260202 15:09:01.859269 15003 raft_consensus.cc:740] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:09:01.868551 15390 consensus_meta.cc:287] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.700s user 0.011s sys 0.000s
I20260202 15:09:01.875464 15302 raft_consensus.cc:697] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:01.876345 15556 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.878546 12689 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:01.865072 15557 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:01.889348 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 92523.3us
I20260202 15:09:01.897087 15302 consensus_queue.cc:237] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:01.901825 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:01.902375 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:01.903141 13362 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEARNER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:01.914530 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:09:01.916471 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 56843.9us
I20260202 15:09:01.918629 12764 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:01.922405 13359 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:09:01.898067 15390 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.899451 14681 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:01.931576 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:01.932756 13362 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:09:01.907814 15568 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:01.898499 15579 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:01.881269 15035 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.889719 15502 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.969933 13354 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 9: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:01.976070 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:01.978557 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:01.979274 12648 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 9.
W20260202 15:09:01.986058 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 82665.5us
I20260202 15:09:01.962544 15579 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:01.987543 12690 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:01.988415 12690 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:01.854317 15555 consensus_queue.cc:237] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:01.990458 12614 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:01.977026 14681 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:01.980368 15568 raft_consensus.cc:697] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:01.965766 15463 consensus_queue.cc:1059] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
W20260202 15:09:02.006311 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 107110us
I20260202 15:09:01.964460 15571 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.125s
I20260202 15:09:02.008430 15549 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:01.989457 15579 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:02.014430 12615 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:01.985249 15155 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260202 15:09:02.001962 15546 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:02.003737 15494 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:01.987043 15407 consensus_queue.cc:237] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.050130 12615 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:01.983115 15260 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.022986 15113 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:02.026280 15569 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260202 15:09:02.026594 15585 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.074s
I20260202 15:09:02.050149 12690 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:02.052510 15414 consensus_meta.cc:287] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.797s user 0.011s sys 0.000s
I20260202 15:09:02.051820 12644 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:02.010988 15549 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:02.015154 13326 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:01.980448 15003 consensus_queue.cc:260] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.018688 15456 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:09:02.058279 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:02.058457 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 47437.6us
I20260202 15:09:02.023881 15546 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:01.990844 15591 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260202 15:09:02.131531 15549 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:02.067541 15414 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.126472 15582 consensus_queue.cc:1048] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.149s
I20260202 15:09:02.176364 15557 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:02.028772 15584 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:02.230346 15035 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:02.179349 15489 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:02.128001 15592 log.cc:927] Time spent T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
W20260202 15:09:02.130328 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 80254.4us
I20260202 15:09:02.170738 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:02.005995 14681 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:09:02.252925 15581 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.011s
I20260202 15:09:02.253566 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.254098 14681 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:09:02.197510 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 288998us
W20260202 15:09:02.128207 15595 log.cc:927] Time spent T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260202 15:09:02.254885 12650 raft_consensus.cc:1275] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:09:02.055847 15567 consensus_queue.cc:1059] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.127s
I20260202 15:09:02.255976 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.256126 14681 consensus_queue.cc:1048] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:09:02.256649 12650 raft_consensus.cc:2393] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 7: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:02.256660 15573 raft_consensus.cc:3055] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:02.257367 15573 raft_consensus.cc:740] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:02.258461 15573 consensus_queue.cc:260] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:02.127234 15599 log.cc:927] Time spent T 4d68e0d1324241f1878c450b36fac925 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260202 15:09:02.057272 15411 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:02.167102 15569 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader election won for term 4
I20260202 15:09:02.261287 15569 raft_consensus.cc:697] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:02.261786 15003 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:02.262396 15569 consensus_queue.cc:237] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.227309 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:02.266382 12721 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:02.169727 15397 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:02.196787 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 169116us
I20260202 15:09:02.214633 15260 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
W20260202 15:09:02.231868 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:02.224498 15390 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:02.161015 15494 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.191164 15606 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260202 15:09:02.247829 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.236052 15546 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:02.253121 12648 raft_consensus.cc:1275] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260202 15:09:02.255410 12689 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:02.256848 12717 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:02.248623 15550 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:02.063077 15463 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:02.254119 12646 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:02.234565 15502 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:02.126502 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260202 15:09:02.235380 15602 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:02.273523 12718 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 9: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:02.240813 15456 raft_consensus.cc:697] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:02.275019 12644 raft_consensus.cc:1275] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260202 15:09:02.297179 12644 pending_rounds.cc:85] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 7
I20260202 15:09:02.297405 12644 pending_rounds.cc:107] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260202 15:09:02.297597 12644 raft_consensus.cc:2889] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:02.277603 12615 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:02.278862 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.299176 12650 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:01.864028 12792 rpcz_store.cc:276] 0202 15:09:00.614320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:00.619029 (+ 4709us) service_pool.cc:225] Handling call
0202 15:09:00.644555 (+ 25526us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:00.644949 (+ 394us) raft_consensus.cc:1534] Early marking committed up to index 4
0202 15:09:00.915354 (+270405us) log.cc:844] Serialized 14 byte log entry
0202 15:09:01.747016 (+831662us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:01.760427 (+ 13411us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:09:01.769579 (+ 9152us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b1800349c60. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea4b1402 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad 0000000000379254 0000000000379099 00007f24ea49469a
0202 15:09:01.774904 (+ 5325us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:01.785009 (+ 10105us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:01.792668 (+ 7659us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21838,"prepare.queue_time_us":316539,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":2676,"raft.queue_time_us":30653,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":4591,"spinlock_wait_cycles":5089152,"thread_start_us":987750,"threads_started":3,"wal-append.queue_time_us":552897}
W20260202 15:09:02.324626 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 47131us
I20260202 15:09:02.327512 12721 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8.
W20260202 15:09:02.320072 15600 log.cc:927] Time spent T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20260202 15:09:02.293763 15411 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:02.330267 12615 leader_election.cc:400] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7.
I20260202 15:09:02.331710 12615 leader_election.cc:403] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260202 15:09:02.334929 15540 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367) with higher term. Message: Invalid argument: T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 6. Current term is 7.
I20260202 15:09:02.337359 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.337862 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.339006 12795 raft_consensus.cc:1275] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260202 15:09:02.339843 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.340801 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:02.341277 12799 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:02.343000 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.344254 12800 raft_consensus.cc:1275] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:09:02.345785 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.346398 12798 raft_consensus.cc:2393] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:02.354849 12799 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:02.275416 15397 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.321921 15605 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:02.298192 15489 consensus_queue.cc:1059] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260202 15:09:02.373777 15173 raft_consensus.cc:3055] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:02.374118 15173 raft_consensus.cc:740] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:02.374764 15173 consensus_queue.cc:260] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.375656 15173 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:02.232438 15568 consensus_queue.cc:237] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:02.359186 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 78793.7us
W20260202 15:09:02.359639 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 109005us
I20260202 15:09:02.367870 12765 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:02.372543 15610 consensus_queue.cc:1059] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20260202 15:09:02.374842 15575 consensus_queue.cc:1059] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20260202 15:09:02.329337 15155 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:02.407923 12690 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:09:02.409749 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 51597.1us
I20260202 15:09:02.411849 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:02.413040 13355 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:09:02.322031 15463 raft_consensus.cc:2749] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:09:02.431051 15456 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:02.432413 15603 log.cc:927] Time spent T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
W20260202 15:09:02.409970 15544 log.cc:927] Time spent T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260202 15:09:02.436823 15417 consensus_queue.cc:237] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:02.437767 15394 consensus_meta.cc:287] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.834s user 0.006s sys 0.004s
I20260202 15:09:02.438318 15394 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:02.416877 12794 raft_consensus.cc:1275] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260202 15:09:02.419291 15609 log.cc:927] Time spent T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260202 15:09:02.423640 15575 consensus_queue.cc:1243] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 6921687ed3814cccb2bd9c606875b9f0 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:09:02.425601 15494 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:02.426941 15590 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.268s user 0.000s sys 0.001s
I20260202 15:09:02.299810 15602 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:02.440186 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:02.440516 15155 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.415445 15585 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:09:02.408650 15549 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.443109 15502 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:02.443763 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.444087 12722 raft_consensus.cc:1275] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260202 15:09:02.445096 15397 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:02.446228 15610 consensus_queue.cc:1243] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:09:02.446614 13355 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEARNER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260202 15:09:02.446893 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.447523 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.414093 15608 raft_consensus.cc:2749] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260202 15:09:02.414433 15540 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:02.358723 15019 consensus_queue.cc:1048] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.091s
W20260202 15:09:02.461402 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 53576.9us
I20260202 15:09:02.483064 12690 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:02.358201 15411 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.460671 13363 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:02.462937 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.478610 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:02.480015 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.480865 15575 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260202 15:09:02.481148 15463 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:09:02.481494 15494 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:02.481835 12615 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:02.463922 15591 raft_consensus.cc:2764] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260202 15:09:02.484128 12724 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:09:02.489754 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.494361 12721 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:02.472342 15585 raft_consensus.cc:697] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:02.492167 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:02.478415 15567 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.015s
I20260202 15:09:02.490490 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.499059 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.504964 12721 raft_consensus.cc:1240] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:09:02.475427 15502 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.497288 15414 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:02.511579 15579 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.519448 15463 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:09:02.519764 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.519788 15618 consensus_queue.cc:1059] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
W20260202 15:09:02.547612 15614 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260202 15:09:02.488071 15608 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.020s
I20260202 15:09:02.550513 12726 raft_consensus.cc:1275] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260202 15:09:02.514998 15585 consensus_queue.cc:237] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.484833 15540 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.511360 15588 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:02.519985 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:02.497022 15494 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:02.565536 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:02.558110 15454 consensus_meta.cc:287] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.690s user 0.004s sys 0.017s
I20260202 15:09:02.569203 12799 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:02.573186 15494 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:09:02.570479 15611 log.cc:927] Time spent T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.077s user 0.004s sys 0.000s
I20260202 15:09:02.580870 15454 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.584692 12797 raft_consensus.cc:1240] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:02.583547 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.588088 12800 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260202 15:09:02.592669 15579 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:02.593725 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:02.595304 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.596376 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.596629 15454 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:02.596925 12798 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
I20260202 15:09:02.597959 15602 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:02.598843 15410 consensus_meta.cc:287] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.755s user 0.004s sys 0.007s
I20260202 15:09:02.599278 15410 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:02.600004 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.600536 12799 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:09:02.603068 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.593597 15378 raft_consensus.cc:1027] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: attempt to promote peer 66b8c715b9e6479e9218b08bd3156380: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260202 15:09:02.605371 15540 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:02.605547 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:02.606037 15549 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:02.606617 15502 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:02.606734 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.607095 15411 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:02.607409 12798 raft_consensus.cc:2393] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 7: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:02.611783 15546 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.613574 15546 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:02.614955 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:02.605438 15113 consensus_meta.cc:287] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.545s user 0.005s sys 0.005s
I20260202 15:09:02.615515 12793 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:09:02.608764 12792 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:09:02.615585 15113 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260202 15:09:02.617514 15564 log.cc:927] Time spent T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.853s user 0.003s sys 0.000s
I20260202 15:09:02.618896 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 419) took 1058 ms. Trace:
I20260202 15:09:02.620930 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1596) took 1987 ms. Trace:
I20260202 15:09:02.626948 12649 raft_consensus.cc:1240] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:02.633302 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.633749 12718 rpcz_store.cc:276] 0202 15:09:01.560773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:01.560872 (+ 99us) service_pool.cc:225] Handling call
0202 15:09:02.618662 (+1057790us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54002ff654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:02.618817 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1358455552}
I20260202 15:09:02.646064 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.634389 12791 rpcz_store.cc:276] 0202 15:09:00.633866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:00.633990 (+ 124us) service_pool.cc:225] Handling call
0202 15:09:00.647075 (+ 13085us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:00.687382 (+ 40307us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:09:01.279457 (+592075us) log.cc:844] Serialized 14 byte log entry
0202 15:09:01.736026 (+456569us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:01.758753 (+ 22727us) log.cc:844] Serialized 26 byte log entry
0202 15:09:01.759270 (+ 517us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:01.759352 (+ 82us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:02.620602 (+861250us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b54001f7cd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:02.620645 (+ 43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:02.620689 (+ 44us) raft_consensus.cc:1685] finished
0202 15:09:02.620769 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:02.620880 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":335985,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":142230,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":203,"spinlock_wait_cycles":91567360,"thread_start_us":1022025,"threads_started":3,"wal-append.queue_time_us":413222}
I20260202 15:09:02.646797 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.656232 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:02.647156 12763 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:02.660944 12646 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:02.660962 12644 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:02.662173 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.656947 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:02.705972 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.707964 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:02.574046 15588 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.710492 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:02.710441 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 208888us
I20260202 15:09:02.710461 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:02.731498 12724 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:02.707373 15634 consensus_queue.cc:1048] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.144s
I20260202 15:09:02.706550 15574 consensus_queue.cc:1048] T f8f84948a93c4dc984e69bfd1f44be23 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260202 15:09:02.712509 12722 raft_consensus.cc:3060] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:02.714951 12643 raft_consensus.cc:2410] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260202 15:09:02.710222 15155 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:02.718235 12642 raft_consensus.cc:2393] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:02.717988 12721 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:02.757195 12724 raft_consensus.cc:2468] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:02.730912 12726 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:02.734128 12646 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
W20260202 15:09:02.736881 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 89786.6us
I20260202 15:09:02.738899 15549 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:02.743781 15588 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:09:02.744719 15635 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260202 15:09:02.732141 12615 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:02.765515 12722 raft_consensus.cc:2468] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:02.766355 12763 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:02.779539 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:02.793736 15155 consensus_queue.cc:1048] T ee143496d46d4a45a2d24f0990e74f44 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260202 15:09:02.794097 12615 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:02.794875 15634 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:02.795332 15634 raft_consensus.cc:697] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:02.796257 15634 consensus_queue.cc:237] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.798828 12615 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:02.801965 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.802469 12798 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:09:02.803865 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:02.803588 12615 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:02.805675 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.806428 12615 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:02.807946 12615 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:02.809701 15575 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260202 15:09:02.809460 12793 raft_consensus.cc:2955] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:09:02.810858 15575 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:09:02.811162 15575 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:02.811465 15575 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:02.798429 15546 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:02.814918 15546 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.816546 15546 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:02.800654 15417 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260202 15:09:02.803061 15574 raft_consensus.cc:2749] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:02.819754 12763 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:02.820438 15574 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:02.820737 15574 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:02.821019 15574 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:02.821198 12763 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:02.822268 15494 raft_consensus.cc:2804] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:02.803834 15113 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:02.822680 15494 raft_consensus.cc:697] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:02.804741 15623 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:02.825089 15623 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.802654 15639 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:02.803128 15588 consensus_queue.cc:1048] T 4d879eebd1924ad8b6f65647f4e73362 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.024s
I20260202 15:09:02.808560 15463 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:02.807611 15567 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:09:02.823513 15494 consensus_queue.cc:237] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.802927 15502 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
W20260202 15:09:02.835286 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:02.847911 15623 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:02.851668 12646 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:09:02.863410 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.872068 15463 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:02.883680 15574 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.884938 15572 raft_consensus.cc:3055] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Stepping down as leader of term 3
I20260202 15:09:02.885448 15572 raft_consensus.cc:740] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:02.886204 15572 consensus_queue.cc:260] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.852273 15502 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.887229 15572 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
I20260202 15:09:02.888502 15502 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:02.865204 15410 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20260202 15:09:02.893757 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 712) took 1089 ms. Trace:
I20260202 15:09:02.894152 13363 rpcz_store.cc:276] 0202 15:09:01.802052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:01.802329 (+ 277us) service_pool.cc:225] Handling call
0202 15:09:01.802619 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:02.483779 (+681160us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:02.890530 (+406751us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b540059e2d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:02.890667 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1512812416}
I20260202 15:09:02.907641 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.910816 12648 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260202 15:09:02.908934 15410 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:02.920346 15575 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.937546 12642 raft_consensus.cc:2393] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 5: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:02.939227 15503 consensus_queue.cc:260] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.942278 15411 consensus_queue.cc:1059] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:09:02.942981 15411 consensus_queue.cc:1243] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260202 15:09:02.950541 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 142679us
I20260202 15:09:02.953698 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.954345 12794 raft_consensus.cc:2393] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:02.916460 15502 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260202 15:09:02.950043 15552 raft_consensus.cc:3055] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:02.960994 15552 raft_consensus.cc:740] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:02.961731 15552 consensus_queue.cc:260] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.962921 15552 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:02.963225 15621 raft_consensus.cc:3055] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:02.963627 15621 raft_consensus.cc:740] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:02.964377 15621 consensus_queue.cc:260] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:09:02.966195 15574 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:02.956960 15575 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:02.931368 15410 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:02.960671 12794 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:09:02.970317 15410 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:02.971997 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 696) took 1166 ms. Trace:
W20260202 15:09:02.975047 13328 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:02.975616 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:02.974499 12720 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1811) took 1412 ms (client timeout 1574 ms). Trace:
I20260202 15:09:02.976435 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:02.976949 12643 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 7
W20260202 15:09:02.976179 12720 rpcz_store.cc:269] 0202 15:09:01.561824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:01.561984 (+ 160us) service_pool.cc:225] Handling call
0202 15:09:02.974366 (+1412382us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b5400158954. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:02.974451 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":430563072}
I20260202 15:09:02.972698 13361 rpcz_store.cc:276] 0202 15:09:01.805617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:01.831171 (+ 25554us) service_pool.cc:225] Handling call
0202 15:09:01.831420 (+ 249us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:01.831591 (+ 171us) raft_consensus.cc:1534] Early marking committed up to index 9
0202 15:09:01.921262 (+ 89671us) log.cc:844] Serialized 14 byte log entry
0202 15:09:02.971449 (+1050187us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b18004e1d80. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea42646a 00007f24f3aac5c9 00007f24ea2e6160 00007f24ea4ae3c4 00007f24ea4ae335 00007f24ed556ce0 00007f24ed569396 00007f24ed579f07 00007f24ed579e88
0202 15:09:02.971626 (+ 177us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:02.971672 (+ 46us) raft_consensus.cc:1645] Marking committed up to 9
0202 15:09:02.971718 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:02.971799 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:02.971931 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16111,"prepare.queue_time_us":161673,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"spinlock_wait_cycles":10070912,"thread_start_us":168417,"threads_started":2,"wal-append.queue_time_us":304406}
I20260202 15:09:02.986637 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:02.987243 12644 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:02.987428 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:02.988516 12794 raft_consensus.cc:2393] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:02.992013 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:02.996922 15642 log.cc:927] Time spent T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.141s user 0.002s sys 0.000s
I20260202 15:09:03.004208 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:03.004904 15559 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:03.005448 15559 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:02.997714 12763 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:09:02.999058 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 986421us
W20260202 15:09:02.999627 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:03.015952 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:03.016631 13363 raft_consensus.cc:3060] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:03.018730 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:03.019634 13358 raft_consensus.cc:2393] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 8: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:03.020153 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:03.020815 13356 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:03.011139 15394 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260202 15:09:03.021487 15394 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:03.021768 15394 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Advancing to term 10
W20260202 15:09:03.029680 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 49017.5us
I20260202 15:09:03.033881 15559 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:03.052222 15173 consensus_queue.cc:1059] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260202 15:09:03.015331 15572 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:03.044459 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:03.054808 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:03.133663 12648 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:03.153573 12794 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:03.079378 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:03.151283 13354 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 869) took 1943 ms (client timeout 1627 ms). Trace:
W20260202 15:09:03.082500 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:03.126287 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 46513.9us
I20260202 15:09:03.087865 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:03.108721 13356 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
W20260202 15:09:03.154592 13354 rpcz_store.cc:269] 0202 15:09:01.207863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:01.216566 (+ 8703us) service_pool.cc:225] Handling call
0202 15:09:03.151031 (+1934465us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b54005ed154. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:03.151175 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1629255424}
I20260202 15:09:03.159659 13363 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8.
I20260202 15:09:03.157158 12649 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:03.057117 12643 raft_consensus.cc:2468] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:03.087090 13358 raft_consensus.cc:1275] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260202 15:09:03.164897 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 680902us
I20260202 15:09:03.165367 12763 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:03.174327 12648 raft_consensus.cc:1240] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:09:03.174409 13359 raft_consensus.cc:1240] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260202 15:09:03.173606 12615 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2
W20260202 15:09:03.174815 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 691750us
I20260202 15:09:03.175304 13363 raft_consensus.cc:1275] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:09:03.176000 12615 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:03.035676 15647 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:03.180459 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:03.126405 15576 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:03.158977 15572 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:03.053287 15559 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:03.183140 13328 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:03.180543 15542 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:03.186439 13357 raft_consensus.cc:1275] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260202 15:09:03.186513 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:03.187070 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:03.191151 13328 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:03.244307 13358 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 9: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:09:03.247709 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:03.175529 15456 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:03.256685 12794 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:09:03.166738 15502 raft_consensus.cc:493] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:03.257447 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:03.154209 15035 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260202 15:09:03.053958 15623 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.132s
I20260202 15:09:03.090757 15584 consensus_queue.cc:1059] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.097s
I20260202 15:09:03.256880 12648 raft_consensus.cc:1240] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:03.081377 15394 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.177313 15410 raft_consensus.cc:2804] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
W20260202 15:09:03.329814 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 82197.9us
I20260202 15:09:03.330451 12649 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 9.
W20260202 15:09:03.330054 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 138982us
I20260202 15:09:03.329473 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:03.337385 13328 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:09:03.337553 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:03.341240 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:03.342610 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 153732us
I20260202 15:09:03.343531 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:03.086783 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:09:03.225445 15572 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:03.174515 15302 consensus_queue.cc:1059] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.162s
I20260202 15:09:03.181021 15113 consensus_queue.cc:1048] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.154s
I20260202 15:09:03.263936 12765 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:03.244612 15446 raft_consensus.cc:2749] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:03.333851 15392 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:03.260648 15588 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:03.365098 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 101279us
I20260202 15:09:03.244984 15542 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:03.190173 15173 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:03.214797 15576 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:03.370620 12688 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:03.380128 12765 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:03.254879 15559 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.261471 15489 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:03.189713 15557 consensus_queue.cc:1048] T 7d28e4999b504df4a53610187731416d P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.175s
I20260202 15:09:03.347067 15390 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:03.334666 15463 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:03.369019 15583 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:03.386087 12765 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:03.386763 15494 raft_consensus.cc:2804] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:03.379325 15574 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:03.383453 15542 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:03.380077 15392 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.363050 15410 raft_consensus.cc:697] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:03.334405 15502 raft_consensus.cc:515] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.259689 15647 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:03.384184 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 135612us
W20260202 15:09:03.407399 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:03.510305 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 124293us
I20260202 15:09:03.464993 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 422) took 1022 ms. Trace:
W20260202 15:09:03.512109 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:03.414978 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 44429us
I20260202 15:09:03.371912 15456 raft_consensus.cc:697] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:03.513451 12649 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260202 15:09:03.406033 13354 raft_consensus.cc:1275] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260202 15:09:03.411243 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:03.515626 12644 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:03.367478 15578 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:03.532785 12719 raft_consensus.cc:1240] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:03.371824 15297 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:03.528378 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:03.524168 15574 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.407934 15570 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:09:03.414919 15648 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.497s user 0.002s sys 0.000s
I20260202 15:09:03.404479 15394 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 10 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:09:03.413836 15650 log.cc:927] Time spent T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.515s user 0.002s sys 0.000s
I20260202 15:09:03.379155 15626 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:03.533620 15558 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:03.488879 15433 raft_consensus.cc:1068] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2: Unable to promote non-voter 66b8c715b9e6479e9218b08bd3156380: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.524725 15636 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:03.607800 12726 raft_consensus.cc:1240] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 3. Current term is 4. Ops: []
I20260202 15:09:03.490093 15550 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:03.611088 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:03.511417 12717 rpcz_store.cc:276] 0202 15:09:02.442220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:02.447968 (+ 5748us) service_pool.cc:225] Handling call
0202 15:09:02.448296 (+ 328us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:02.448740 (+ 444us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:09:02.449045 (+ 305us) log.cc:844] Serialized 14 byte log entry
0202 15:09:02.886568 (+437523us) spinlock_profiling.cc:243] Waited 2.48 ms on lock 0x7b1800325ba0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea4b1437 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad 0000000000379254 0000000000379099 00007f24ea49469a
0202 15:09:03.095113 (+208545us) spinlock_profiling.cc:243] Waited 971 us on lock 0x7b1800325ba0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea42646a 00007f24f3aac5c9 00007f24f14348ef 00007f24ea4b1451 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad
0202 15:09:03.256449 (+161336us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:03.256983 (+ 534us) log.cc:844] Serialized 52 byte log entry
0202 15:09:03.257119 (+ 136us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:09:03.260479 (+ 3360us) log.cc:844] Serialized 14 byte log entry
0202 15:09:03.365156 (+104677us) log.cc:844] Serialized 14 byte log entry
0202 15:09:03.365305 (+ 149us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:03.365363 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:03.464797 (+ 99434us) raft_consensus.cc:1685] finished
0202 15:09:03.464816 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:03.464939 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":798986,"prepare.queue_time_us":5410,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":7325,"raft.queue_time_us":709011,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":37969,"spinlock_wait_cycles":178590208,"thread_start_us":140518,"threads_started":1,"wal-append.queue_time_us":290820}
I20260202 15:09:03.405710 15576 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:03.418159 15608 raft_consensus.cc:2804] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:03.602128 15494 raft_consensus.cc:697] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:03.384707 15546 consensus_queue.cc:1048] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.221s
I20260202 15:09:03.395401 15588 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.533363 15559 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:03.518819 15463 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.637421 12720 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260202 15:09:03.651916 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:03.557118 15297 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:03.518689 15583 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:03.604391 15570 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:09:03.630149 15637 log.cc:927] Time spent T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260202 15:09:03.636317 15550 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.619710 15302 consensus_queue.cc:1059] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20260202 15:09:03.644734 15666 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:03.645381 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:03.626878 15626 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.641995 15546 consensus_queue.cc:1243] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:09:03.519521 15489 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.630560 15636 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
W20260202 15:09:03.622990 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 374153us
W20260202 15:09:03.706264 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:03.690312 15503 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:03.692497 15621 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:09:03.701854 15641 log.cc:927] Time spent T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.853s user 0.003s sys 0.000s
I20260202 15:09:03.642304 15573 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:03.626822 15581 consensus_queue.cc:1059] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
W20260202 15:09:03.707669 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 370301us
I20260202 15:09:03.708958 12763 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:03.724226 12797 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:09:03.658255 15297 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:03.655356 15608 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:03.688197 15570 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:03.631058 15558 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:03.715224 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 373243us
I20260202 15:09:03.601943 15578 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:03.707023 15648 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260202 15:09:03.725349 12643 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:09:03.621769 15410 consensus_queue.cc:237] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:03.705518 15666 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:03.852953 15643 log.cc:927] Time spent T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.795s user 0.004s sys 0.000s
I20260202 15:09:03.936710 15584 consensus_queue.cc:1048] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.020s
I20260202 15:09:03.956310 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:03.956972 15666 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:03.957424 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1693) took 1329 ms. Trace:
I20260202 15:09:03.957783 12641 rpcz_store.cc:276] 0202 15:09:02.627697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:02.627870 (+ 173us) service_pool.cc:225] Handling call
0202 15:09:02.628253 (+ 383us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:02.633236 (+ 4983us) raft_consensus.cc:1534] Early marking committed up to index 2
0202 15:09:02.645435 (+ 12199us) log.cc:844] Serialized 14 byte log entry
0202 15:09:02.811951 (+166516us) spinlock_profiling.cc:243] Waited 926 us on lock 0x7b1800164500. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe5da 00007f24ed5696aa
0202 15:09:02.872544 (+ 60593us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:02.874260 (+ 1716us) log.cc:844] Serialized 26 byte log entry
0202 15:09:02.875094 (+ 834us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:02.884604 (+ 9510us) log.cc:844] Serialized 14 byte log entry
0202 15:09:02.885049 (+ 445us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:02.885145 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:03.513698 (+628553us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:09:03.957217 (+443519us) raft_consensus.cc:1685] finished
0202 15:09:03.957239 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:03.957336 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26109,"prepare.queue_time_us":19933,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":3138,"raft.queue_time_us":17255,"raft.run_cpu_time_us":1346,"raft.run_wall_time_us":11182,"spinlock_wait_cycles":28388992,"thread_start_us":60376,"threads_started":1,"wal-append.queue_time_us":208981}
I20260202 15:09:03.958999 14681 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260202 15:09:03.852687 15644 log.cc:927] Time spent T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.950s user 0.003s sys 0.000s
I20260202 15:09:03.761003 15618 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:03.960551 15618 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:03.960875 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1916) took 1159 ms. Trace:
I20260202 15:09:03.960934 15618 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:03.961306 12799 rpcz_store.cc:276] 0202 15:09:02.801028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:02.801143 (+ 115us) service_pool.cc:225] Handling call
0202 15:09:02.806037 (+ 4894us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:02.808896 (+ 2859us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:02.808923 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:02.815181 (+ 6258us) log.cc:844] Serialized 26 byte log entry
0202 15:09:02.876736 (+ 61555us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:02.885204 (+ 8468us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:02.885267 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:03.139319 (+254052us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b4800281dd8. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f2478e21 00007f24f246feb7 00007f24f246fd4a 00007f24f2566f7f 00007f24f2566c47 00007f24f257b26c 00007f24f257b218 00007f24f257b198 00007f24f257b153 00007f24f257a3fc 00007f24f14b55b6
0202 15:09:03.706253 (+566934us) spinlock_profiling.cc:243] Waited 41 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:09:03.960733 (+254480us) raft_consensus.cc:1685] finished
0202 15:09:03.960750 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:03.960830 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3282,"raft.run_cpu_time_us":2598,"raft.run_wall_time_us":329864,"spinlock_wait_cycles":134797440,"thread_start_us":61201,"threads_started":1,"wal-append.queue_time_us":86839}
I20260202 15:09:03.962505 15578 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:03.879002 15392 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:03.888520 15463 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:03.895620 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 109149us
I20260202 15:09:03.760658 12615 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:03.966234 13328 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:03.967027 15463 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election won for term 5
W20260202 15:09:03.834477 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:03.969908 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1912) took 1169 ms. Trace:
I20260202 15:09:03.889124 15647 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:03.912506 15663 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:03.879370 15502 leader_election.cc:290] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:03.967450 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 206844us
W20260202 15:09:03.974076 15646 log.cc:927] Time spent T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 1.105s user 0.003s sys 0.005s
W20260202 15:09:03.999970 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:04.004125 13328 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:03.993405 12793 rpcz_store.cc:276] 0202 15:09:02.800101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:02.800209 (+ 108us) service_pool.cc:225] Handling call
0202 15:09:02.802058 (+ 1849us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:02.808367 (+ 6309us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:09:02.836611 (+ 28244us) log.cc:844] Serialized 14 byte log entry
0202 15:09:02.915385 (+ 78774us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:02.928350 (+ 12965us) log.cc:844] Serialized 52 byte log entry
0202 15:09:02.928543 (+ 193us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:02.928595 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:02.928685 (+ 90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:03.414350 (+485665us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:09:03.969531 (+555181us) raft_consensus.cc:1685] finished
0202 15:09:03.969658 (+ 127us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:03.969816 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5748,"raft.queue_time_us":5779,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":19294,"spinlock_wait_cycles":58162432,"thread_start_us":67285,"threads_started":1,"wal-append.queue_time_us":70515}
I20260202 15:09:04.019033 15297 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:04.000219 15572 consensus_meta.cc:287] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.633s user 0.006s sys 0.003s
I20260202 15:09:04.025147 13328 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:04.030519 12690 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.036s sys 0.050s
I20260202 15:09:03.763846 12722 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260202 15:09:04.031448 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:04.031733 12689 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.038s sys 0.049s
I20260202 15:09:03.712041 15636 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:03.796955 15035 consensus_queue.cc:1059] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.241s
I20260202 15:09:04.022419 15463 raft_consensus.cc:697] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:03.795011 15494 consensus_queue.cc:237] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:03.711764 15456 consensus_queue.cc:237] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:04.396626 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:04.086315 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:04.407423 12797 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:04.022291 15584 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260202 15:09:04.406296 12689 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.023s user 0.044s sys 0.059s
W20260202 15:09:04.413290 15552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.000s sys 0.020s
W20260202 15:09:04.142835 15639 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.012s sys 0.034s
W20260202 15:09:04.420498 15639 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.902s user 0.014s sys 0.043s
I20260202 15:09:04.420548 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.222452 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1911) took 1422 ms. Trace:
I20260202 15:09:04.359287 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 426) took 1915 ms. Trace:
I20260202 15:09:04.442519 12792 rpcz_store.cc:276] 0202 15:09:02.799857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:02.799962 (+ 105us) service_pool.cc:225] Handling call
0202 15:09:02.800230 (+ 268us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:02.800521 (+ 291us) raft_consensus.cc:1534] Early marking committed up to index 7
0202 15:09:02.800544 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:02.805019 (+ 4475us) log.cc:844] Serialized 26 byte log entry
0202 15:09:02.868088 (+ 63069us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:09:02.868137 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:02.868218 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:03.973520 (+1105302us) spinlock_profiling.cc:243] Waited 60.8 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:09:04.222143 (+248623us) raft_consensus.cc:1685] finished
0202 15:09:04.222169 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:04.222383 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1891,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":68,"spinlock_wait_cycles":139734912,"thread_start_us":36059,"threads_started":1,"wal-append.queue_time_us":36775}
I20260202 15:09:04.359023 12645 raft_consensus.cc:1275] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260202 15:09:04.029589 15578 raft_consensus.cc:2749] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260202 15:09:04.363087 15573 consensus_meta.cc:287] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.639s user 0.005s sys 0.005s
I20260202 15:09:03.857533 15567 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:04.081765 13326 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:04.477177 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:04.497434 12721 raft_consensus.cc:2376] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7. Re-sending same reply.
I20260202 15:09:04.480234 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.497985 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:04.483340 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 458217us
I20260202 15:09:04.412609 15584 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:03.760460 15608 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
W20260202 15:09:04.416761 15552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s user 0.000s sys 0.020s
I20260202 15:09:03.794318 15626 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:03.792932 15173 raft_consensus.cc:2749] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:04.087965 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:04.024791 15666 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.396s
I20260202 15:09:04.085402 15502 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:04.517251 15679 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260202 15:09:04.415613 15670 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.499454 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:04.534014 13328 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:04.535588 13328 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:04.363749 15634 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:03.794869 15662 consensus_queue.cc:1048] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.816s
I20260202 15:09:04.365499 15550 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:04.331080 14681 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:04.501688 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 419985us
I20260202 15:09:04.442620 12725 rpcz_store.cc:276] 0202 15:09:02.443388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:02.449491 (+ 6103us) service_pool.cc:225] Handling call
0202 15:09:02.472706 (+ 23215us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:02.477268 (+ 4562us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:09:02.477524 (+ 256us) log.cc:844] Serialized 14 byte log entry
0202 15:09:02.896109 (+418585us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b180048e800. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe5da 00007f24ed5696aa
0202 15:09:03.329691 (+433582us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:03.329735 (+ 44us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:03.329774 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:04.358982 (+1029208us) spinlock_profiling.cc:243] Waited 4.35 ms on lock 0x7b540019b754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:04.359053 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:04.359225 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":407228,"prepare.queue_time_us":115,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":1589,"raft.queue_time_us":377768,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":8554,"spinlock_wait_cycles":13787904,"thread_start_us":55906,"threads_started":1,"wal-append.queue_time_us":524109,"wal-append.run_cpu_time_us":1121,"wal-append.run_wall_time_us":1011250}
I20260202 15:09:04.497732 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:04.249270 15669 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.021694 15663 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.232380 15574 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:04.505316 12719 raft_consensus.cc:1217] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 3.5->[5.6-8.7] Dedup: 5.6->[8.7-8.7]
I20260202 15:09:04.185645 15566 raft_consensus.cc:3055] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Stepping down as leader of term 8
I20260202 15:09:04.505513 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.505751 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.554133 12724 raft_consensus.cc:2468] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 9.
I20260202 15:09:04.515327 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:04.484119 12690 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.100s user 0.042s sys 0.065s
W20260202 15:09:04.555626 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 1.21212e+06us
I20260202 15:09:04.505825 15584 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Advancing to term 9
W20260202 15:09:04.519325 15503 consensus_meta.cc:287] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.805s user 0.009s sys 0.000s
I20260202 15:09:04.524261 12799 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:09:04.517323 15666 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:04.537756 12650 raft_consensus.cc:606] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Received request to transfer leadership to TS 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:09:04.508518 15573 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.502800 15572 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:04.541958 15653 log.cc:927] Time spent T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 1.553s user 0.004s sys 0.011s
I20260202 15:09:04.551194 12719 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 8. Current term is 9. Ops: [5.6-8.7]
I20260202 15:09:04.540391 15297 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:04.514919 15567 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.540966 15502 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:04.570109 13360 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.080s user 0.025s sys 0.074s
I20260202 15:09:04.488543 15674 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260202 15:09:04.627912 12650 raft_consensus.cc:614] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260202 15:09:04.628525 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 72310.9us
I20260202 15:09:04.543646 15578 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:04.505300 15671 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260202 15:09:04.630131 13360 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.140s user 0.030s sys 0.082s
I20260202 15:09:04.541401 15392 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:04.539398 15670 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.545975 15558 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:04.553464 15566 raft_consensus.cc:740] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:04.545931 15673 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:04.637075 15582 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.634449 15666 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:04.661098 15566 consensus_queue.cc:260] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.662118 15566 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
W20260202 15:09:04.670020 12615 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.009s sys 0.050s
W20260202 15:09:04.670382 12615 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.009s sys 0.050s
I20260202 15:09:04.552129 15550 consensus_queue.cc:1059] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.092s
W20260202 15:09:04.662890 12613 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.001s sys 0.053s
W20260202 15:09:04.692134 12613 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s user 0.001s sys 0.053s
I20260202 15:09:04.635902 15578 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:09:04.696455 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:04.546316 15634 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:04.552378 14681 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.663067 15503 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.547225 15463 consensus_queue.cc:237] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:04.692792 15672 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.699394 15173 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:04.692749 15680 raft_consensus.cc:493] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:04.689527 15682 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:04.733976 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 254796us
W20260202 15:09:04.696813 15677 log.cc:927] Time spent T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.284s user 0.000s sys 0.001s
I20260202 15:09:04.736311 12649 raft_consensus.cc:1275] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260202 15:09:04.633468 15573 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:04.708143 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:04.704548 12615 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:04.742190 12689 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:04.744988 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:04.733047 15552 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.743005 15680 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:04.735850 15621 consensus_meta.cc:287] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.017s user 0.010s sys 0.000s
I20260202 15:09:04.747327 15621 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.746868 15552 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:09:04.747805 15621 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.749328 15552 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:04.575436 15669 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.746893 12719 raft_consensus.cc:1275] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260202 15:09:04.702912 15578 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:04.752622 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1703) took 1917 ms. Trace:
W20260202 15:09:04.756255 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 210918us
W20260202 15:09:04.756783 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 212477us
I20260202 15:09:04.786070 13326 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:04.786804 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.787976 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.743146 15688 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:04.791729 12722 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:09:04.699820 15662 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.689291 15582 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.799350 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.801359 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:04.807185 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.778774 12647 rpcz_store.cc:276] 0202 15:09:02.834938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:02.835040 (+ 102us) service_pool.cc:225] Handling call
0202 15:09:02.851181 (+ 16141us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:02.866552 (+ 15371us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:02.866595 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:02.890232 (+ 23637us) log.cc:844] Serialized 52 byte log entry
0202 15:09:02.985854 (+ 95622us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:02.985874 (+ 20us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:02.985913 (+ 39us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:04.160717 (+1174804us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b440002304c. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea3c0005 00007f24ea3c0a22 00007f24ed10a57b 00007f24ed105e4f 00007f24ed0f9c0e 00007f24ed0f8f56 00007f24ed0f8b78 00007f24ed1025c1 00007f24ed102579 00007f24ed102509
0202 15:09:04.744382 (+583665us) raft_consensus.cc:1685] finished
0202 15:09:04.744408 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:04.752554 (+ 8146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3461,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":67,"spinlock_wait_cycles":27609856,"thread_start_us":95332,"threads_started":1,"wal-append.queue_time_us":98218}
W20260202 15:09:04.788600 12614 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.46.196:36191 timed out after 2.140s (SENT) [suppressed 5 similar messages]
W20260202 15:09:04.815807 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 609114us
I20260202 15:09:04.818128 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:04.818912 12649 raft_consensus.cc:2393] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:04.742403 15503 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.830528 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.832015 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:04.793998 15674 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader election won for term 9
W20260202 15:09:04.828567 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 135348us
I20260202 15:09:04.743796 15173 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:04.851248 15573 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:04.817693 15686 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.830440 15621 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:04.741204 15672 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:04.900074 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 111524us
I20260202 15:09:04.832530 15685 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:04.852069 15678 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:04.924064 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 181851us
I20260202 15:09:04.770118 15682 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:04.926208 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 98154.6us
I20260202 15:09:04.859328 15663 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:04.928311 12689 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:04.808688 15662 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:04.863993 15568 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:04.930238 12765 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:04.897560 15417 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:04.795238 15688 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.939291 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 880) took 1770 ms. Trace:
I20260202 15:09:04.891482 15692 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:04.905380 15572 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:04.941588 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:04.941289 13363 rpcz_store.cc:276] 0202 15:09:03.168379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.181581 (+ 13202us) service_pool.cc:225] Handling call
0202 15:09:03.249086 (+ 67505us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:03.367700 (+118614us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:09:03.383938 (+ 16238us) log.cc:844] Serialized 14 byte log entry
0202 15:09:03.646411 (+262473us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b18003baa60. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea4b12d3 00007f24ea4b36a1 00007f24ea4b3659 00007f24ea4b35e9 00007f24ea4b35b1 00007f24ea4b28ad 0000000000379254 0000000000379099 00007f24ea49469a
0202 15:09:03.981230 (+334819us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:04.022863 (+ 41633us) log.cc:844] Serialized 26 byte log entry
0202 15:09:04.025359 (+ 2496us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:04.025387 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:04.025440 (+ 53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:04.938561 (+913121us) raft_consensus.cc:1685] finished
0202 15:09:04.938648 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:04.939235 (+ 587us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":119030,"prepare.queue_time_us":16860,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":142675,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":29294,"spinlock_wait_cycles":12441472,"thread_start_us":469381,"threads_started":1,"wal-append.queue_time_us":899174}
I20260202 15:09:04.935457 15663 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:04.863808 15574 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260202 15:09:04.861917 15695 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.086s
W20260202 15:09:04.952136 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:04.933207 15502 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:09:04.940560 15570 consensus_meta.cc:287] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.090s user 0.007s sys 0.003s
W20260202 15:09:04.961383 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:04.961807 12765 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:04.944958 15666 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:04.934772 15690 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:04.940397 15567 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:04.978523 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:04.979110 12719 raft_consensus.cc:1275] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260202 15:09:04.983487 12717 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:09:04.986580 15684 log.cc:927] Time spent T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.290s user 0.001s sys 0.000s
I20260202 15:09:04.986495 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:04.891705 15686 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.961866 15692 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.930930 15678 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:04.930529 15685 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:04.944370 15417 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:04.944851 15568 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:05.106299 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 143750us
I20260202 15:09:04.987264 12793 raft_consensus.cc:2376] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6. Re-sending same reply.
I20260202 15:09:05.153415 15173 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:05.172338 15417 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:04.975580 14681 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:04.964068 15572 raft_consensus.cc:2749] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260202 15:09:05.018180 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:05.173331 15678 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:05.174324 14681 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.050441 15567 consensus_queue.cc:1048] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20260202 15:09:05.174765 14681 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:05.176095 15670 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:05.176385 14681 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:05.176797 15676 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:05.178299 15688 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:05.179519 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1693) took 1468 ms. Trace:
W20260202 15:09:05.180217 15542 consensus_meta.cc:287] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 1.646s user 0.010s sys 0.000s
I20260202 15:09:05.180605 15542 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:05.181134 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:05.181476 12644 raft_consensus.cc:1240] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:05.181922 15414 raft_consensus.cc:3055] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:09:05.182258 15414 raft_consensus.cc:740] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:09:05.182986 15618 consensus_meta.cc:287] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.221s user 0.010s sys 0.002s
I20260202 15:09:05.130676 12724 raft_consensus.cc:1240] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260202 15:09:05.184165 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1719) took 1429 ms. Trace:
I20260202 15:09:05.044134 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:04.987356 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:05.186313 12797 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:09:05.129591 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:05.045295 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 187080us
I20260202 15:09:05.076805 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 701) took 2075 ms. Trace:
I20260202 15:09:05.023032 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:05.194466 12717 raft_consensus.cc:2393] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:05.017510 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:05.173096 15569 raft_consensus.cc:3055] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:05.179833 12642 rpcz_store.cc:276] 0202 15:09:03.711075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.711199 (+ 124us) service_pool.cc:225] Handling call
0202 15:09:05.178841 (+1467642us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:09:05.179013 (+ 172us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:05.179198 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:05.179234 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:05.179287 (+ 53us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:05.179323 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:05.179375 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:05.179473 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4403072}
W20260202 15:09:04.986824 15683 log.cc:927] Time spent T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
I20260202 15:09:04.987596 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:05.198563 12799 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
W20260202 15:09:05.051555 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 89814us
W20260202 15:09:05.126327 15640 log.cc:927] Time spent T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 2.276s user 0.011s sys 0.029s
I20260202 15:09:05.185009 12644 rpcz_store.cc:276] 0202 15:09:03.754665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:04.064797 (+310132us) service_pool.cc:225] Handling call
0202 15:09:04.144753 (+ 79956us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:05.181814 (+1037061us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:05.183981 (+ 2167us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400299854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:05.184116 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2383832832}
I20260202 15:09:04.968921 15695 consensus_queue.cc:1243] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260202 15:09:05.192516 13361 rpcz_store.cc:276] 0202 15:09:03.001895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.006346 (+ 4451us) service_pool.cc:225] Handling call
0202 15:09:03.023066 (+ 16720us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:03.034664 (+ 11598us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:03.034696 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:03.043742 (+ 9046us) log.cc:844] Serialized 26 byte log entry
0202 15:09:04.640159 (+1596417us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b18004e1ba0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471 00007f24ed51a73f
0202 15:09:04.848683 (+208524us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:09:04.848784 (+ 101us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:04.848852 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:05.076554 (+227702us) raft_consensus.cc:1685] finished
0202 15:09:05.076573 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:05.076753 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8130,"raft.queue_time_us":12663,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":17135,"spinlock_wait_cycles":32458496,"thread_start_us":208351,"threads_started":1,"wal-append.queue_time_us":1862353}
I20260202 15:09:04.981726 15621 raft_consensus.cc:2749] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260202 15:09:05.207177 13363 raft_consensus.cc:1240] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 10 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 9. Current term is 10. Ops: []
I20260202 15:09:05.051061 15703 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:05.214254 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:05.189528 15608 consensus_meta.cc:287] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.674s user 0.007s sys 0.002s
I20260202 15:09:05.124874 15709 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:04.981735 15634 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:05.164781 15712 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.050047 15689 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.213697 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:05.075013 15503 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:05.128813 15260 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:04.987161 15666 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.215754 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:05.215227 12647 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:09:05.121114 15682 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:05.129935 15693 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.075389 15702 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:05.216835 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:05.216394 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:05.218290 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:05.257329 12644 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:05.257418 13359 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:05.189492 15173 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.220409 15705 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.221612 13354 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260202 15:09:05.217715 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:05.197054 15569 raft_consensus.cc:740] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:05.197078 15567 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:05.183013 15414 consensus_queue.cc:260] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:05.222882 13328 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:05.221356 15669 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:04.964529 15663 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:05.229439 12793 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
I20260202 15:09:05.229785 12718 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:05.230876 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:05.075045 15699 consensus_queue.cc:1048] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.250s
I20260202 15:09:05.242862 13358 raft_consensus.cc:2410] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260202 15:09:05.211948 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:05.290922 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:09:05.291558 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:05.291510 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:04.975940 15570 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:05.183372 15618 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:05.220403 15672 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:05.295238 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:05.298892 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:05.302373 12615 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:05.302462 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:05.262987 15673 consensus_meta.cc:287] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.572s user 0.008s sys 0.000s
I20260202 15:09:05.306413 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:05.233147 15542 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:05.219233 15703 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:05.232249 15572 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.085s
I20260202 15:09:05.231889 15417 consensus_queue.cc:1048] T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.086s
I20260202 15:09:05.287128 15260 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:05.222091 15608 raft_consensus.cc:515] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:09:05.234233 15689 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:05.226764 15709 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:05.232885 15712 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:05.294922 12764 connection.cc:232] RPC call timeout handler was delayed by -1.0428217699998186s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2237s for a response
W20260202 15:09:05.432883 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:05.372283 13360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 703) took 2321 ms. Trace:
I20260202 15:09:05.567271 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 498) took 2054 ms. Trace:
I20260202 15:09:05.568383 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:05.371006 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:05.849781 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 547428us
W20260202 15:09:05.398926 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 176119us
I20260202 15:09:05.852339 12724 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:05.853257 13328 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
W20260202 15:09:05.853874 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:05.413913 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:05.367884 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 72498.7us
W20260202 15:09:05.569859 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:05.856176 13359 raft_consensus.cc:2410] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260202 15:09:05.398658 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:05.850322 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1692) took 1049 ms. Trace:
I20260202 15:09:05.854303 13328 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:05.855697 12765 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:05.857445 12718 rpcz_store.cc:276] 0202 15:09:04.801070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:04.801184 (+ 114us) service_pool.cc:225] Handling call
0202 15:09:05.850271 (+1049087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:05.851739 12646 rpcz_store.cc:276] 0202 15:09:03.512561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.524302 (+ 11741us) service_pool.cc:225] Handling call
0202 15:09:03.605506 (+ 81204us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:03.704728 (+ 99222us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:03.704772 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:03.704828 (+ 56us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:09:03.704874 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:05.566945 (+1862071us) spinlock_profiling.cc:243] Waited 99 ms on lock 0x7b54002a8d54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:05.567009 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:05.567199 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":227668224}
I20260202 15:09:05.850639 12797 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
W20260202 15:09:05.343024 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:05.633574 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1696) took 1662 ms. Trace:
I20260202 15:09:05.851287 13360 rpcz_store.cc:276] 0202 15:09:03.002642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.013214 (+ 10572us) service_pool.cc:225] Handling call
0202 15:09:03.017614 (+ 4400us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:03.035740 (+ 18126us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:09:03.035770 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:03.087241 (+ 51471us) log.cc:844] Serialized 26 byte log entry
0202 15:09:03.490132 (+402891us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b18004e1c60. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471 00007f24ed51a73f
0202 15:09:04.639572 (+1149440us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:04.639612 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:04.639668 (+ 56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:05.323791 (+684123us) raft_consensus.cc:1685] finished
0202 15:09:05.323809 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:05.323999 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16749,"raft.queue_time_us":18136,"spinlock_wait_cycles":85284992,"thread_start_us":1149271,"threads_started":1,"wal-append.queue_time_us":1612500}
I20260202 15:09:05.859418 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:05.861467 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:05.863272 12646 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:05.864558 13354 raft_consensus.cc:1240] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:05.861682 12649 rpcz_store.cc:276] 0202 15:09:03.971184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:04.847551 (+876367us) service_pool.cc:225] Handling call
0202 15:09:04.892525 (+ 44974us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:04.892851 (+ 326us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:04.892884 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:04.904065 (+ 11181us) log.cc:844] Serialized 26 byte log entry
0202 15:09:05.190084 (+286019us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:09:05.190135 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:05.190215 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:05.633332 (+443117us) raft_consensus.cc:1685] finished
0202 15:09:05.633359 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:05.633506 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10896,"raft.queue_time_us":154937,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":63219,"thread_start_us":245968,"threads_started":1,"wal-append.queue_time_us":357616}
W20260202 15:09:05.645551 15636 consensus_meta.cc:287] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.225s user 0.007s sys 0.002s
W20260202 15:09:05.640738 15584 consensus_meta.cc:287] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.948s user 0.008s sys 0.000s
W20260202 15:09:05.612500 15463 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.000s sys 0.010s
I20260202 15:09:05.872774 12615 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:05.304837 15690 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:05.302548 15662 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:05.415920 15694 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.418686 15414 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:05.567041 15678 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:05.949945 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:06.050778 12642 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
W20260202 15:09:05.414039 15392 consensus_meta.cc:287] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.772s user 0.009s sys 0.000s
I20260202 15:09:05.876363 12765 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:05.617909 15685 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:05.612125 15621 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:09:05.627381 15710 log.cc:927] Time spent T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.391s user 0.000s sys 0.000s
W20260202 15:09:05.870795 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 135417us
W20260202 15:09:06.049106 15456 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.005s sys 0.021s
I20260202 15:09:05.951563 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:05.952186 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:05.417671 15692 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:05.659914 15583 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:05.296036 15173 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.004243 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:06.125437 12646 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
W20260202 15:09:06.037429 15724 log.cc:927] Time spent T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.276s user 0.000s sys 0.000s
I20260202 15:09:06.041786 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1828) took 1564 ms. Trace:
I20260202 15:09:05.997365 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1830) took 1518 ms. Trace:
W20260202 15:09:06.038430 15723 log.cc:927] Time spent T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260202 15:09:06.036396 15716 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:05.853116 15720 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:05.956555 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:06.127949 12722 rpcz_store.cc:276] 0202 15:09:04.478631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:04.507204 (+ 28573us) service_pool.cc:225] Handling call
0202 15:09:04.507448 (+ 244us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:04.792434 (+284986us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:05.997164 (+1204730us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b540019a354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:05.997314 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":563455232}
I20260202 15:09:06.134044 13355 raft_consensus.cc:2393] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 10: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:05.417799 15503 consensus_queue.cc:1059] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.551s
I20260202 15:09:06.019546 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1670) took 2471 ms. Trace:
I20260202 15:09:06.135690 12720 rpcz_store.cc:276] 0202 15:09:03.527312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.527428 (+ 116us) service_pool.cc:225] Handling call
0202 15:09:03.527655 (+ 227us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:03.653180 (+125525us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:05.997934 (+2344754us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54002fec54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:05.998171 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":240494976}
I20260202 15:09:05.288281 15666 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.039701 15417 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:05.860754 15673 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:05.618472 15700 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:05.657987 15566 consensus_meta.cc:287] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.995s user 0.008s sys 0.002s
I20260202 15:09:05.832729 15695 raft_consensus.cc:493] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.370941 15701 consensus_queue.cc:1059] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.601s
I20260202 15:09:05.646221 15713 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:05.741375 15669 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:05.860261 15698 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:09:05.950928 15463 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.871s user 0.000s sys 0.016s
I20260202 15:09:05.293952 15702 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:05.997617 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:06.176259 13359 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:06.039929 15689 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.077484 15621 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:05.951321 15494 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:05.896346 15729 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:05.956243 15578 consensus_meta.cc:287] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.165s user 0.011s sys 0.000s
I20260202 15:09:05.919739 15676 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:06.176707 15574 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.017s sys 0.024s
W20260202 15:09:05.967741 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 95047.8us
I20260202 15:09:05.950012 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:05.859716 15665 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:06.213848 15709 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.214190 15665 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:06.214609 15665 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:06.215040 15489 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:05.997392 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 143134us
I20260202 15:09:06.219132 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:06.219640 12643 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
W20260202 15:09:06.220090 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1724) took 2297 ms (client timeout 1632 ms). Trace:
W20260202 15:09:06.220360 12643 rpcz_store.cc:269] 0202 15:09:03.922205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.958069 (+ 35864us) service_pool.cc:225] Handling call
0202 15:09:06.218999 (+2260930us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:09:06.220013 (+ 1014us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3308800}
I20260202 15:09:06.221254 13328 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:05.955519 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:06.221415 15580 consensus_queue.cc:1048] T 9e78bd8179024ae0aca19878b91342a9 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:09:06.222047 13361 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8.
I20260202 15:09:06.222905 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:06.223817 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:05.999989 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:06.224267 15735 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:05.302373 15693 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } }
I20260202 15:09:06.224886 15673 consensus_queue.cc:1059] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:09:06.224937 15735 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:06.225714 15673 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:06.226203 15693 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:09:06.067888 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 191589us
I20260202 15:09:06.226972 12765 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:06.028889 15662 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:06.227730 15662 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:06.228127 15662 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:06.229192 13328 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:06.229847 15673 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:06.230182 15673 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:06.230470 15673 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Advancing to term 9
W20260202 15:09:06.210597 15574 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.546s user 0.017s sys 0.025s
I20260202 15:09:05.303503 15705 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:05.998250 15725 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:05.950866 15584 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:06.075417 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:06.242831 12642 raft_consensus.cc:2468] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:06.243393 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1740) took 1034 ms. Trace:
I20260202 15:09:06.243708 12642 rpcz_store.cc:276] 0202 15:09:05.209008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.051402 (+842394us) service_pool.cc:225] Handling call
0202 15:09:06.243330 (+191928us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:05.966135 15636 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.058465 15726 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:06.070055 15392 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260202 15:09:06.076220 15456 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.661s user 0.005s sys 0.022s
I20260202 15:09:06.212093 15618 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:05.976096 15690 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.137027 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:06.178478 15695 raft_consensus.cc:515] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:06.075091 15540 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:06.127774 12723 rpcz_store.cc:276] 0202 15:09:04.477855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:04.478007 (+ 152us) service_pool.cc:225] Handling call
0202 15:09:04.503036 (+ 25029us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:04.828977 (+325941us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:04.829383 (+ 406us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:04.849836 (+ 20453us) log.cc:844] Serialized 26 byte log entry
0202 15:09:05.223650 (+373814us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:05.223694 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:05.223775 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:06.041543 (+817768us) raft_consensus.cc:1685] finished
0202 15:09:06.041571 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:06.041719 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":232557,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":44370,"thread_start_us":608123,"threads_started":2,"wal-append.queue_time_us":859244}
I20260202 15:09:06.078024 15572 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:06.127384 15722 raft_consensus.cc:493] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:06.153910 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 297533us
I20260202 15:09:06.196825 15494 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.067930 15678 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:06.157732 12641 raft_consensus.cc:2410] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260202 15:09:06.158910 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:06.128413 15732 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:06.177201 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:06.143895 15720 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.181882 13326 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:05.950234 15508 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:06.051697 15694 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:06.151772 15716 raft_consensus.cc:697] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:06.184996 12800 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260202 15:09:05.656708 15686 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:06.213713 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:06.224491 12649 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Advancing to term 10
I20260202 15:09:05.317236 15567 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:06.225276 15727 log.cc:927] Time spent T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.177s user 0.000s sys 0.000s
I20260202 15:09:06.211144 15676 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:05.648162 15569 consensus_queue.cc:260] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.231292 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 435) took 1251 ms. Trace:
I20260202 15:09:06.221254 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:06.153198 15417 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:06.232308 12615 leader_election.cc:336] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Timed out: RequestConsensusVote RPC to 127.11.46.194:44367 timed out after 1.574s (SENT)
I20260202 15:09:06.224282 13361 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
W20260202 15:09:06.234719 12644 consensus_meta.cc:287] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.948s user 0.010s sys 0.000s
I20260202 15:09:06.157701 15700 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.178113 15669 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:06.238363 15573 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.306s user 0.043s sys 0.081s
W20260202 15:09:06.487581 15573 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.561s user 0.043s sys 0.081s
I20260202 15:09:06.258251 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:06.488080 12642 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 7
I20260202 15:09:06.488173 15690 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.488561 15669 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.490206 15621 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:06.067386 15454 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:06.490972 15454 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:06.492029 15700 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.492713 15454 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:06.137439 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:06.203006 15568 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:06.493438 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1700) took 1280 ms. Trace:
I20260202 15:09:06.485843 15173 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.493732 12646 rpcz_store.cc:276] 0202 15:09:05.213129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.126054 (+912925us) service_pool.cc:225] Handling call
0202 15:09:06.493387 (+367333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:06.494786 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:06.495381 15738 log.cc:927] Time spent T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.191s user 0.000s sys 0.000s
I20260202 15:09:06.495976 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:06.496513 12646 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:06.497189 15695 leader_election.cc:290] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.497341 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:06.497864 12646 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:06.177966 15713 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:06.499058 15665 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.516438 15713 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.516861 15665 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.517135 15582 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:06.231490 15260 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.245209 15570 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.155586 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:06.203482 15578 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:06.264501 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 92283.1us
I20260202 15:09:06.268702 12793 raft_consensus.cc:1240] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:06.270491 12720 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:09:06.273100 12689 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:06.254196 15712 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.204101 15729 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:06.183085 15698 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.482045 12644 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:06.289045 12794 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260202 15:09:06.304800 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:06.295096 12796 raft_consensus.cc:1275] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260202 15:09:06.475066 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:06.482897 12615 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:06.251591 15725 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:06.498577 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:06.482003 12725 rpcz_store.cc:276] 0202 15:09:04.979399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:04.979520 (+ 121us) service_pool.cc:225] Handling call
0202 15:09:05.131049 (+151529us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:05.131359 (+ 310us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:05.131392 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:05.150903 (+ 19511us) log.cc:844] Serialized 26 byte log entry
0202 15:09:05.236643 (+ 85740us) raft_consensus.cc:1645] Marking committed up to 9
0202 15:09:05.305392 (+ 68749us) log.cc:844] Serialized 14 byte log entry
0202 15:09:05.744068 (+438676us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:05.744144 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:06.231118 (+486974us) raft_consensus.cc:1685] finished
0202 15:09:06.231135 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:06.231241 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":592,"prepare.queue_time_us":80322,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":90207,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":17578,"spinlock_wait_cycles":269762048,"thread_start_us":361571,"threads_started":2,"wal-append.queue_time_us":543370}
I20260202 15:09:06.502509 12643 raft_consensus.cc:2393] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
W20260202 15:09:06.337121 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:06.380111 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:06.304217 13356 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
W20260202 15:09:06.505170 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 323322us
I20260202 15:09:06.482059 15508 raft_consensus.cc:493] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:06.519896 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 730) took 1126 ms. Trace:
I20260202 15:09:06.241115 15703 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057)
I20260202 15:09:06.267866 15663 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:06.261200 15737 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.345s
I20260202 15:09:06.550233 12642 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
W20260202 15:09:07.202761 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:06.591657 13355 raft_consensus.cc:1275] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEARNER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:09:06.621377 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 881) took 3452 ms. Trace:
I20260202 15:09:06.609018 13360 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
W20260202 15:09:06.710894 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 370053us
W20260202 15:09:07.204609 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1731) took 2012 ms (client timeout 1511 ms). Trace:
I20260202 15:09:07.195775 13358 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260202 15:09:06.269104 15540 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:07.206041 12649 consensus_meta.cc:287] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.724s user 0.010s sys 0.000s
W20260202 15:09:07.057508 13359 consensus_meta.cc:287] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.876s user 0.006s sys 0.002s
I20260202 15:09:07.206218 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 731) took 1845 ms. Trace:
I20260202 15:09:07.206884 13359 raft_consensus.cc:2410] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260202 15:09:07.206956 15670 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:07.207468 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 737) took 1446 ms. Trace:
I20260202 15:09:07.207794 13359 rpcz_store.cc:276] 0202 15:09:05.760825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.898150 (+137325us) service_pool.cc:225] Handling call
0202 15:09:07.207417 (+1309267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.207808 15670 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:07.207593 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 936553us
I20260202 15:09:06.552641 15569 raft_consensus.cc:3060] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:07.210611 15670 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:06.885159 12647 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 504) took 1947 ms (client timeout 1530 ms). Trace:
I20260202 15:09:07.211226 15670 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260202 15:09:07.211505 12647 rpcz_store.cc:269] 0202 15:09:04.937884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:04.941378 (+ 3494us) service_pool.cc:225] Handling call
0202 15:09:06.884924 (+1943546us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54000799d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:06.885051 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":620888960}
I20260202 15:09:07.212785 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:07.214675 12647 raft_consensus.cc:2410] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260202 15:09:07.205446 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 729) took 1845 ms. Trace:
I20260202 15:09:06.467864 15735 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:07.205799 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1742) took 1033 ms. Trace:
I20260202 15:09:06.303578 15702 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.217545 12642 rpcz_store.cc:276] 0202 15:09:06.172275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.257941 (+ 85666us) service_pool.cc:225] Handling call
0202 15:09:07.202977 (+945036us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.206786 15743 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:07.208105 15626 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:07.219045 15743 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.220759 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1727) took 2480 ms. Trace:
I20260202 15:09:07.206629 13358 rpcz_store.cc:276] 0202 15:09:05.360472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.486154 (+1125682us) service_pool.cc:225] Handling call
0202 15:09:07.195490 (+709336us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:07.205985 (+ 10495us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.206157 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.222259 15698 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.200350 12763 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:09:07.071017 13362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 712) took 3988 ms (client timeout 1592 ms). Trace:
I20260202 15:09:07.203588 13357 rpcz_store.cc:276] 0202 15:09:03.169075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.256752 (+ 87677us) service_pool.cc:225] Handling call
0202 15:09:03.339896 (+ 83144us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:03.603736 (+263840us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:09:05.740844 (+2137108us) spinlock_profiling.cc:243] Waited 5.93 ms on lock 0x7b18003bab20. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4ae335 00007f24f0f59317 00007f24ed569627 00007f24ed578d3f 00007f24ed578cb8
0202 15:09:06.046153 (+305309us) log.cc:844] Serialized 14 byte log entry
0202 15:09:06.243286 (+197133us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:06.266668 (+ 23382us) log.cc:844] Serialized 26 byte log entry
0202 15:09:06.266871 (+ 203us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:09:06.266918 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:06.621057 (+354139us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b540016c1d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:06.621111 (+ 54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:06.621163 (+ 52us) raft_consensus.cc:1685] finished
0202 15:09:06.621180 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:06.621318 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":42901,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":27986,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":17561,"spinlock_wait_cycles":619606272,"thread_start_us":348227,"threads_started":2,"wal-append.queue_time_us":198795}
I20260202 15:09:07.224135 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1732) took 2031 ms. Trace:
I20260202 15:09:07.224457 12647 rpcz_store.cc:276] 0202 15:09:05.192663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.212505 (+2019842us) service_pool.cc:225] Handling call
0202 15:09:07.224073 (+ 11568us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.224704 15582 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:07.225133 15582 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:07.225461 15582 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:07.226434 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1729) took 2439 ms. Trace:
I20260202 15:09:06.520902 15572 raft_consensus.cc:697] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
W20260202 15:09:07.204979 12644 rpcz_store.cc:269] 0202 15:09:05.192044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.202750 (+ 10706us) service_pool.cc:225] Handling call
0202 15:09:07.204535 (+2001785us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.228178 15735 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:07.228992 15735 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:07.229812 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:07.230121 15572 consensus_queue.cc:237] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:07.231410 15407 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:06.276597 15685 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:07.232158 15685 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.232672 12725 raft_consensus.cc:1240] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:07.232918 15582 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.216256 15663 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260202 15:09:06.806255 15749 log.cc:927] Time spent T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.167s user 0.000s sys 0.004s
I20260202 15:09:07.223479 12763 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:06.781103 15713 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:07.236133 15721 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:07.236505 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:07.236589 15721 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:07.237491 12725 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:07.105674 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1822) took 2691 ms. Trace:
I20260202 15:09:07.216862 13355 rpcz_store.cc:276] 0202 15:09:05.359989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.485619 (+1125630us) service_pool.cc:225] Handling call
0202 15:09:06.591409 (+105790us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.203298 (+611889us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.205387 (+ 2089us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.239794 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:07.219552 15626 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:07.220933 15743 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:06.621014 15636 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:07.248557 15626 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:07.208670 15729 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:06.619256 15705 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.209120 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:07.249922 13359 raft_consensus.cc:2410] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260202 15:09:07.250747 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:07.240633 13355 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:07.251247 13359 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:07.251511 15725 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.211881 15670 consensus_queue.cc:1243] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260202 15:09:07.222440 12642 rpcz_store.cc:276] 0202 15:09:04.739997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.219917 (+2479920us) service_pool.cc:225] Handling call
0202 15:09:07.220231 (+ 314us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.220410 (+ 179us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.220442 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.220497 (+ 55us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:09:07.220539 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.220600 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.220702 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.251981 13361 rpcz_store.cc:276] 0202 15:09:05.360270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.485790 (+1125520us) service_pool.cc:225] Handling call
0202 15:09:06.486096 (+ 306us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:06.486319 (+ 223us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:09:06.486342 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:06.486377 (+ 35us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:06.486406 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:06.486455 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:06.486559 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.226838 12647 rpcz_store.cc:276] 0202 15:09:04.787140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.225483 (+2438343us) service_pool.cc:225] Handling call
0202 15:09:07.225716 (+ 233us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.226201 (+ 485us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.226220 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.226253 (+ 33us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:07.226276 (+ 23us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.226319 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.226396 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":74,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":63}
I20260202 15:09:07.253357 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1733) took 2060 ms. Trace:
I20260202 15:09:07.253504 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1734) took 2060 ms. Trace:
I20260202 15:09:07.252030 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:07.253858 12647 rpcz_store.cc:276] 0202 15:09:05.193461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.252955 (+2059494us) service_pool.cc:225] Handling call
0202 15:09:07.253143 (+ 188us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.253286 (+ 143us) raft_consensus.cc:1534] Early marking committed up to index 7
0202 15:09:07.253304 (+ 18us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.253333 (+ 29us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:09:07.253358 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.253400 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.253470 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.254492 13355 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:09:07.254583 12649 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 10.
I20260202 15:09:07.238888 15721 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:09:07.255085 12649 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 501) took 2317 ms (client timeout 2937 ms). Trace:
I20260202 15:09:07.255350 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1735) took 2061 ms. Trace:
W20260202 15:09:07.255455 12649 rpcz_store.cc:269] 0202 15:09:04.937216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.895637 (+958421us) service_pool.cc:225] Handling call
0202 15:09:07.255026 (+1359389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.256003 12763 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:07.255741 12647 rpcz_store.cc:276] 0202 15:09:05.193680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.254760 (+2061080us) service_pool.cc:225] Handling call
0202 15:09:07.254989 (+ 229us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.255124 (+ 135us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.255143 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.255179 (+ 36us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:07.255201 (+ 22us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.255241 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.255313 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.256731 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1736) took 2059 ms. Trace:
I20260202 15:09:07.229348 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 503) took 2291 ms. Trace:
I20260202 15:09:07.256915 12763 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:06.269376 15718 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:07.257170 12644 rpcz_store.cc:276] 0202 15:09:04.937672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.228488 (+2290816us) service_pool.cc:225] Handling call
0202 15:09:07.228799 (+ 311us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.228950 (+ 151us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.228973 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.229009 (+ 36us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:07.229038 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.229097 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.229299 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.236466 15713 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.257930 12763 leader_election.cc:304] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:07.258826 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1739) took 2050 ms. Trace:
I20260202 15:09:07.259011 15721 raft_consensus.cc:2804] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:07.259510 15721 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:07.259716 15713 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:07.260011 15721 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:07.238921 12726 rpcz_store.cc:276] 0202 15:09:04.435787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:04.435904 (+ 117us) service_pool.cc:225] Handling call
0202 15:09:04.493757 (+ 57853us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:04.796184 (+302427us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:09:04.796221 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:04.827550 (+ 31329us) log.cc:844] Serialized 26 byte log entry
0202 15:09:04.976529 (+148979us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:04.976564 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.059261 (+2082697us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b540033ac54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:07.059338 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:07.059454 (+ 116us) raft_consensus.cc:1685] finished
0202 15:09:07.059474 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.105596 (+ 46122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7630,"raft.queue_time_us":150456,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":20108,"spinlock_wait_cycles":694654848,"thread_start_us":85301,"threads_started":1,"wal-append.queue_time_us":332703}
I20260202 15:09:07.247527 15417 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:07.260763 15417 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:07.259204 12644 rpcz_store.cc:276] 0202 15:09:05.208768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.258089 (+2049321us) service_pool.cc:225] Handling call
0202 15:09:07.258359 (+ 270us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.258520 (+ 161us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.258544 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.258583 (+ 39us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:07.258620 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.258670 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.258778 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.261262 15417 raft_consensus.cc:697] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:07.262122 12725 raft_consensus.cc:2468] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 6.
I20260202 15:09:07.262144 15297 raft_consensus.cc:2749] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:06.608489 12800 raft_consensus.cc:1240] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
W20260202 15:09:06.626350 15414 consensus_meta.cc:287] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.624s user 0.006s sys 0.002s
I20260202 15:09:07.252030 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:07.263734 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:07.263861 15692 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:07.264276 12800 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:07.264441 13359 raft_consensus.cc:1240] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:09:07.265154 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:06.810607 15747 log.cc:927] Time spent T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.186s user 0.004s sys 0.002s
I20260202 15:09:07.265826 12800 raft_consensus.cc:2410] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260202 15:09:06.503722 15732 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:07.264371 15692 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:06.609172 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 156273us
I20260202 15:09:07.266413 15732 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:06.610102 15567 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:07.216377 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 984001us
I20260202 15:09:07.267117 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1623) took 1093 ms. Trace:
I20260202 15:09:07.267169 12688 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:07.253669 12642 rpcz_store.cc:276] 0202 15:09:05.193202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.252773 (+2059571us) service_pool.cc:225] Handling call
0202 15:09:07.252985 (+ 212us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.253132 (+ 147us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.253152 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.253182 (+ 30us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:07.253205 (+ 23us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.253246 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.253317 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:09:06.526808 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:07.268344 12795 raft_consensus.cc:1217] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 3.4->[4.5-6.6] Dedup: 4.5->[6.6-6.6]
I20260202 15:09:07.267551 12798 rpcz_store.cc:276] 0202 15:09:06.174203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.174405 (+ 202us) service_pool.cc:225] Handling call
0202 15:09:06.190715 (+ 16310us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:06.210879 (+ 20164us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:06.210901 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:06.211371 (+ 470us) log.cc:844] Serialized 52 byte log entry
0202 15:09:06.304724 (+ 93353us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:09:06.304771 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:06.304845 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:07.266922 (+962077us) raft_consensus.cc:1685] finished
0202 15:09:07.266957 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.267068 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19740,"raft.queue_time_us":39024,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":90300,"threads_started":1,"wal-append.queue_time_us":410553}
I20260202 15:09:07.267782 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1928) took 1131 ms. Trace:
I20260202 15:09:07.269088 12642 raft_consensus.cc:1240] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260202 15:09:07.269761 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1698) took 2057 ms. Trace:
I20260202 15:09:07.269006 12795 raft_consensus.cc:1240] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: [4.5-6.6]
I20260202 15:09:07.269991 12798 raft_consensus.cc:1240] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: []
I20260202 15:09:06.624585 15662 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:07.267551 12615 connection.cc:232] RPC call timeout handler was delayed by -1.3464384980002251s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2121s for a response
I20260202 15:09:07.271140 12798 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260202 15:09:07.269392 12797 rpcz_store.cc:276] 0202 15:09:06.135931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.136050 (+ 119us) service_pool.cc:225] Handling call
0202 15:09:06.140311 (+ 4261us) raft_consensus.cc:1471] Updating replica for 3 ops
0202 15:09:06.171406 (+ 31095us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:06.171447 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0202 15:09:06.180482 (+ 9035us) log.cc:844] Serialized 616 byte log entry
0202 15:09:06.319394 (+138912us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:09:06.489567 (+170173us) log.cc:844] Serialized 14 byte log entry
0202 15:09:07.198074 (+708507us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.202328 (+ 4254us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:07.267609 (+ 65281us) raft_consensus.cc:1685] finished
0202 15:09:07.267626 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.267737 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":151541,"prepare.queue_time_us":170014,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":760527,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":12228,"thread_start_us":79005,"threads_started":1,"wal-append.queue_time_us":405687}
W20260202 15:09:07.226845 13362 rpcz_store.cc:269] 0202 15:09:03.082778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.070802 (+3988024us) service_pool.cc:212] Skipping call since client already timed out
0202 15:09:07.070941 (+ 139us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260202 15:09:07.270644 15670 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:07.272688 15662 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:07.272644 15670 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:07.273069 13362 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260202 15:09:07.273178 12615 connection.cc:232] RPC call timeout handler was delayed by -1.2820927359998677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
I20260202 15:09:06.288872 15722 raft_consensus.cc:515] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:07.273814 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.231477 15735 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:07.275508 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 509) took 1076 ms. Trace:
I20260202 15:09:06.465670 15584 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.268054 15414 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:07.263012 15411 raft_consensus.cc:2804] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:07.278728 12643 rpcz_store.cc:276] 0202 15:09:06.199066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.221077 (+ 22011us) service_pool.cc:225] Handling call
0202 15:09:07.275457 (+1054380us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.279302 15411 raft_consensus.cc:697] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:07.279935 12643 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:07.280066 15411 consensus_queue.cc:237] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:07.280438 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1701) took 2066 ms. Trace:
I20260202 15:09:07.280882 12643 rpcz_store.cc:276] 0202 15:09:05.214369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.279577 (+2065208us) service_pool.cc:225] Handling call
0202 15:09:07.279796 (+ 219us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.280251 (+ 455us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.280397 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.282184 12643 raft_consensus.cc:1275] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260202 15:09:07.275604 15489 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:07.283082 12646 raft_consensus.cc:2410] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260202 15:09:07.283703 13357 raft_consensus.cc:1240] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260202 15:09:07.283901 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1703) took 1422 ms. Trace:
I20260202 15:09:07.284444 15578 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.284379 12643 rpcz_store.cc:276] 0202 15:09:05.861047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.281620 (+1420573us) service_pool.cc:225] Handling call
0202 15:09:07.281867 (+ 247us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.282519 (+ 652us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.282684 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:06.805723 15720 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.621709 15446 raft_consensus.cc:3055] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:07.274351 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.285534 15446 raft_consensus.cc:740] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:07.285970 12643 raft_consensus.cc:1240] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:07.286119 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.275020 13362 raft_consensus.cc:1240] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:07.275652 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1706) took 1007 ms. Trace:
I20260202 15:09:07.287376 12723 rpcz_store.cc:276] 0202 15:09:06.268306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.268516 (+ 210us) service_pool.cc:225] Handling call
0202 15:09:07.275599 (+1007083us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.268054 12688 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:07.280476 15722 leader_election.cc:290] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:06.610128 15676 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:07.275722 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1626) took 1016 ms. Trace:
I20260202 15:09:07.270062 12642 rpcz_store.cc:276] 0202 15:09:05.212779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.268594 (+2055815us) service_pool.cc:225] Handling call
0202 15:09:07.268872 (+ 278us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:07.269556 (+ 684us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.269721 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.291639 12642 raft_consensus.cc:1275] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260202 15:09:07.292296 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1705) took 1430 ms. Trace:
I20260202 15:09:07.286247 15446 consensus_queue.cc:260] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.291036 12796 rpcz_store.cc:276] 0202 15:09:06.259312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.287411 (+ 28099us) service_pool.cc:225] Handling call
0202 15:09:06.294678 (+ 7267us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.198786 (+904108us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.275674 (+ 76888us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.257054 12649 rpcz_store.cc:276] 0202 15:09:05.197745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.256136 (+2058391us) service_pool.cc:225] Handling call
0202 15:09:07.256347 (+ 211us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.256488 (+ 141us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.256508 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.256542 (+ 34us) raft_consensus.cc:1645] Marking committed up to 8
0202 15:09:07.256570 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.256612 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.256690 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.296241 12688 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:07.289862 13362 raft_consensus.cc:1275] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 8 LEARNER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260202 15:09:07.290077 12791 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4.
I20260202 15:09:07.274655 15670 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.286684 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1704) took 1425 ms. Trace:
I20260202 15:09:07.296990 12643 rpcz_store.cc:276] 0202 15:09:05.861295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.285579 (+1424284us) service_pool.cc:225] Handling call
0202 15:09:07.285779 (+ 200us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.286360 (+ 581us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.286641 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.298013 12615 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:07.299980 12800 raft_consensus.cc:1275] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260202 15:09:07.298200 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1619) took 1161 ms. Trace:
I20260202 15:09:07.300923 12791 rpcz_store.cc:276] 0202 15:09:06.136723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.136824 (+ 101us) service_pool.cc:225] Handling call
0202 15:09:07.298148 (+1161324us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.292845 12642 rpcz_store.cc:276] 0202 15:09:05.861606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.291194 (+1429588us) service_pool.cc:225] Handling call
0202 15:09:07.291422 (+ 228us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.292061 (+ 639us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.292244 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:06.616076 15736 raft_consensus.cc:3055] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:06.307775 15666 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.068544 15733 raft_consensus.cc:3055] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 LEADER]: Stepping down as leader of term 9
I20260202 15:09:07.311558 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 507) took 1130 ms. Trace:
I20260202 15:09:06.551505 15695 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260202 15:09:07.311887 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 506) took 1130 ms. Trace:
I20260202 15:09:07.192801 15608 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.191982 15494 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:07.277033 15508 raft_consensus.cc:3060] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:07.337633 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.286723 15585 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260202 15:09:07.349174 12642 rpcz_store.cc:276] 0202 15:09:06.181577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.310477 (+1128900us) service_pool.cc:225] Handling call
0202 15:09:07.310874 (+ 397us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.311085 (+ 211us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.311116 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.311165 (+ 49us) raft_consensus.cc:1645] Marking committed up to 9
0202 15:09:07.311204 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.311265 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.311374 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.349982 12688 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:07.314529 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.350303 12643 rpcz_store.cc:276] 0202 15:09:06.181314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.311113 (+1129799us) service_pool.cc:225] Handling call
0202 15:09:07.311396 (+ 283us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.311594 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.311623 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.311671 (+ 48us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:07.311705 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.311748 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.311836 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.305415 15722 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:07.352885 15733 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:07.353444 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:07.354395 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1706) took 1490 ms. Trace:
I20260202 15:09:07.354655 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:07.367233 12791 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:09:07.295506 15680 leader_election.cc:290] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:07.262151 15417 consensus_queue.cc:237] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:07.344889 12792 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.002s sys 0.011s
I20260202 15:09:07.276000 15407 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:07.363569 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:07.378939 12643 raft_consensus.cc:2468] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:07.310946 15736 raft_consensus.cc:740] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:07.278975 15414 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:07.368388 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:07.387073 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:07.387804 12642 raft_consensus.cc:2393] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 8: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:07.276077 15626 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.310544 15720 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:07.353968 15670 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:06.264549 15726 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.334438 15567 raft_consensus.cc:2749] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:07.312002 15663 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:07.373456 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:07.377851 12792 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s user 0.002s sys 0.011s
I20260202 15:09:07.358798 12800 raft_consensus.cc:1275] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260202 15:09:07.437312 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.383304 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:09:07.384018 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:07.395387 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:07.440357 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:07.363163 15446 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:07.368119 12649 rpcz_store.cc:276] 0202 15:09:05.863989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.294683 (+1430694us) service_pool.cc:225] Handling call
0202 15:09:07.353850 (+ 59167us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.354048 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.354091 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.354139 (+ 48us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:07.354176 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.354234 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.354340 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:07.355646 15705 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.024s
I20260202 15:09:07.458853 15705 consensus_queue.cc:1243] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 6921687ed3814cccb2bd9c606875b9f0 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260202 15:09:07.369529 15722 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.394794 15414 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:07.432513 15685 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:07.433336 15736 consensus_queue.cc:260] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:07.371271 15695 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:07.366770 15608 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:07.294260 15716 consensus_queue.cc:237] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:07.378118 15674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.447s user 0.020s sys 0.007s
I20260202 15:09:07.441342 12720 raft_consensus.cc:2393] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 9: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:09:07.426679 15410 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.003s sys 0.024s
I20260202 15:09:07.441718 15663 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.443502 15718 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:07.397698 15720 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:07.445480 15692 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:09:07.471952 15674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.545s user 0.022s sys 0.010s
I20260202 15:09:07.446290 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:07.475806 15673 consensus_meta.cc:287] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 1.245s user 0.011s sys 0.000s
I20260202 15:09:07.448181 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1624) took 1254 ms. Trace:
I20260202 15:09:07.478487 15718 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:07.480036 15720 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:07.484016 12795 rpcz_store.cc:276] 0202 15:09:06.193628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.193798 (+ 170us) service_pool.cc:225] Handling call
0202 15:09:07.268171 (+1074373us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:07.270223 (+ 2052us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.448009 (+177786us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001cc0d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:07.448127 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2457237760}
I20260202 15:09:07.485195 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:07.485754 12795 raft_consensus.cc:2368] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7.
I20260202 15:09:07.486402 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:07.453227 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1710) took 1085 ms. Trace:
I20260202 15:09:07.493196 12795 raft_consensus.cc:2468] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:09:07.493129 12646 rpcz_store.cc:276] 0202 15:09:06.368319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.498362 (+130043us) service_pool.cc:225] Handling call
0202 15:09:07.452967 (+954605us) spinlock_profiling.cc:243] Waited 83.8 ms on lock 0x7b54002a9754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:07.453160 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":192686848}
I20260202 15:09:07.454869 15694 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:07.450640 15407 leader_election.cc:290] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:07.450595 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:07.455904 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:07.435137 15733 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:07.456861 15687 log.cc:927] Time spent T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 2.664s user 0.000s sys 0.002s
I20260202 15:09:07.457559 15671 raft_consensus.cc:3055] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Stepping down as leader of term 5
W20260202 15:09:07.370162 15690 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.753s user 0.011s sys 0.020s
I20260202 15:09:07.466151 13328 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:07.467232 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:07.504817 12649 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:09:07.372828 15712 consensus_queue.cc:1059] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260202 15:09:07.434521 15678 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:07.515887 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.516834 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.517670 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.518077 12720 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 9.
I20260202 15:09:07.471815 12615 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:07.519516 15626 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:07.519894 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 48150.6us
I20260202 15:09:07.522607 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:07.522789 15694 raft_consensus.cc:2804] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:07.523643 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:07.524765 12791 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260202 15:09:07.524780 15694 raft_consensus.cc:697] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:07.525770 15694 consensus_queue.cc:237] T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:07.526683 13360 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44942 (request call id 732) took 2148 ms (client timeout 2057 ms). Trace:
W20260202 15:09:07.527092 13360 rpcz_store.cc:269] 0202 15:09:05.378579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.872232 (+493653us) service_pool.cc:225] Handling call
0202 15:09:07.526539 (+1654307us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54005d2d54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:07.526628 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1117874432}
I20260202 15:09:07.527726 15722 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:07.528031 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:07.528645 13360 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:07.533887 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:07.490682 15720 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:07.474345 15410 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.004s sys 0.024s
I20260202 15:09:07.394634 15729 consensus_queue.cc:1048] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.071s
I20260202 15:09:07.474684 12763 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:07.462608 15508 raft_consensus.cc:515] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:07.504362 15690 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.887s user 0.012s sys 0.021s
I20260202 15:09:07.476744 12689 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:07.395612 15743 consensus_queue.cc:1059] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20260202 15:09:07.445480 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.481734 12642 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:07.445509 15726 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:07.471215 15608 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:07.449146 15540 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:07.473896 15695 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:07.489593 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1695) took 3577 ms (client timeout 3295 ms). Trace:
I20260202 15:09:07.490360 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1720) took 3704 ms. Trace:
I20260202 15:09:07.491307 12648 raft_consensus.cc:1240] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 9. Current term is 10. Ops: []
W20260202 15:09:07.494428 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:07.438459 12643 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:07.498193 12793 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1929) took 1362 ms. Trace:
I20260202 15:09:07.481392 15674 raft_consensus.cc:697] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 9 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:07.501834 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:07.501942 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:07.503720 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:07.508625 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:07.372990 15725 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260202 15:09:07.534979 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:07.537528 12797 raft_consensus.cc:1240] T d88e30bfa6e641749640dde238a2a8f8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:07.537617 13359 raft_consensus.cc:1275] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:09:07.492316 15673 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:07.648317 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:08.062433 12722 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.003s sys 0.017s
I20260202 15:09:08.081831 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:58672 (request call id 24) took 4988 ms. Trace:
W20260202 15:09:07.794533 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:07.775918 12614 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.003s sys 0.027s
W20260202 15:09:07.809386 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:07.694474 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:07.539603 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:08.011729 13355 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.026s sys 0.026s
W20260202 15:09:07.540238 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 300296us
I20260202 15:09:07.748979 12723 raft_consensus.cc:2393] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 9: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:09:07.681138 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:07.667132 12798 raft_consensus.cc:1240] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
W20260202 15:09:07.860687 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:08.084602 12647 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.015s sys 0.002s
W20260202 15:09:08.096935 12647 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s user 0.015s sys 0.003s
W20260202 15:09:08.085306 12641 rpcz_store.cc:269] 0202 15:09:03.912071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.959718 (+ 47647us) service_pool.cc:225] Handling call
0202 15:09:05.180990 (+1221272us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:09:07.489386 (+2308396us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b540008a2d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:07.489513 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2248304768}
I20260202 15:09:08.099233 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:08.085119 12792 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.003s sys 0.014s
W20260202 15:09:08.100199 12792 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.655s user 0.003s sys 0.015s
I20260202 15:09:08.085335 12800 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:08.085515 12646 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:09:08.085726 12720 raft_consensus.cc:2468] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 9.
W20260202 15:09:08.087880 12614 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.841s user 0.010s sys 0.049s
I20260202 15:09:08.086722 12793 rpcz_store.cc:276] 0202 15:09:06.136212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.144203 (+ 7991us) service_pool.cc:225] Handling call
0202 15:09:06.170224 (+ 26021us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.215268 (+1045044us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.497934 (+282666us) spinlock_profiling.cc:243] Waited 98.3 ms on lock 0x7b5400116f54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:07.498121 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":226059008}
W20260202 15:09:08.104457 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 1.42102e+06us
W20260202 15:09:08.106680 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.107511 12688 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:09:08.107780 12614 connection.cc:232] RPC call timeout handler was delayed by -1.3486713800000416s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
I20260202 15:09:08.085148 12645 rpcz_store.cc:276] 0202 15:09:03.786285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:04.492076 (+705791us) service_pool.cc:225] Handling call
0202 15:09:04.506647 (+ 14571us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:04.751510 (+244863us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:04.751550 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:04.751603 (+ 53us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:04.751650 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.490140 (+2738490us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54000127d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:07.490196 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:07.490291 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":562614016}
W20260202 15:09:08.119556 12614 connection.cc:232] RPC call timeout handler was delayed by -1.2361421069999778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1888s for a response
W20260202 15:09:08.119915 12614 connection.cc:232] RPC call timeout handler was delayed by -1.1849250640000264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
I20260202 15:09:07.502453 15671 raft_consensus.cc:740] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:09:08.087596 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 610860us
I20260202 15:09:08.116783 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1692) took 4386 ms. Trace:
I20260202 15:09:08.125268 12689 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:08.125113 12648 rpcz_store.cc:276] 0202 15:09:03.710480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.710623 (+ 143us) service_pool.cc:225] Handling call
0202 15:09:07.490859 (+3780236us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b5800002498. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2528af0 00007f24f2513459 00007f24f2512f02 00007f24eba9fd34
0202 15:09:07.491101 (+ 242us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.096815 (+605714us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.096980 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4608768}
W20260202 15:09:08.118364 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.118431 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:08.085868 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:08.127723 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:09:08.120198 12614 connection.cc:232] RPC call timeout handler was delayed by -1.1142515150002055s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
W20260202 15:09:08.086519 12722 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s user 0.004s sys 0.018s
I20260202 15:09:08.089979 12791 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
W20260202 15:09:08.129704 12614 connection.cc:232] RPC call timeout handler was delayed by -1.0017299270000422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21400000000000002s for a response
I20260202 15:09:08.086652 12725 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
W20260202 15:09:08.090204 13355 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.834s user 0.031s sys 0.028s
W20260202 15:09:08.087466 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 612826us
I20260202 15:09:08.089416 13358 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:09:08.089224 13360 raft_consensus.cc:2468] T 9592591cd602455aa52fdc1d84e48b63 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
W20260202 15:09:08.120486 12718 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.924s user 0.000s sys 0.013s
W20260202 15:09:08.132923 12718 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.936s user 0.000s sys 0.014s
I20260202 15:09:07.507624 15733 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Advancing to term 10
I20260202 15:09:08.086745 12650 rpcz_store.cc:276] 0202 15:09:03.086514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:03.086650 (+ 136us) service_pool.cc:225] Handling call
0202 15:09:08.074949 (+4988299us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400280d54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed5594d3 00007f24f251713a 00007f24ebab2934 00007f24ebab28c2 00007f24ebab27f1 00007f24ebab276c 00007f24ebab1a22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:08.075051 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2738813824}
I20260202 15:09:08.137636 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:08.137852 12795 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:08.138198 12720 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:08.138947 12614 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.141737 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:07.537458 15718 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.147845 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.144387 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:08.048553 15756 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:08.147773 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:07.990648 15446 raft_consensus.cc:670] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } [suppressed 1 similar messages]
I20260202 15:09:08.121994 15662 raft_consensus.cc:2749] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260202 15:09:08.144771 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:08.147431 12644 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.003s sys 0.013s
W20260202 15:09:08.170779 12644 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.816s user 0.003s sys 0.013s
I20260202 15:09:08.170886 12649 raft_consensus.cc:2468] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:09:08.149886 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:08.083721 15716 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.003s sys 0.007s
I20260202 15:09:08.171622 12646 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 3.
W20260202 15:09:08.172254 13356 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 890) took 2932 ms (client timeout 2158 ms). Trace:
I20260202 15:09:08.172993 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:08.172696 13356 rpcz_store.cc:269] 0202 15:09:05.227598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.235562 (+ 7964us) service_pool.cc:225] Handling call
0202 15:09:08.159827 (+2924265us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b54006090d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:08.159955 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1040814464}
I20260202 15:09:08.175702 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:08.176306 12645 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:08.150899 15726 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:08.134783 15407 consensus_queue.cc:1059] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260202 15:09:08.152021 15729 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:08.039680 15720 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.152721 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:08.152662 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.053776 15663 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.153679 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:08.117586 15574 raft_consensus.cc:2804] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:08.108822 15712 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:08.198374 15574 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:08.198444 15712 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:08.198858 15574 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:08.198932 15712 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:08.097195 15608 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:08.156842 12791 raft_consensus.cc:2410] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260202 15:09:08.202651 15582 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.203295 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.204258 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.205163 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.206130 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:08.206969 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.207835 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.208827 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.209825 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:08.210268 12791 raft_consensus.cc:2376] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 4. Re-sending same reply.
I20260202 15:09:08.211648 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 736) took 2464 ms. Trace:
I20260202 15:09:08.212069 13357 rpcz_store.cc:276] 0202 15:09:05.747331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.224012 (+1476681us) service_pool.cc:225] Handling call
0202 15:09:07.224311 (+ 299us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:07.284054 (+ 59743us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.211358 (+927304us) spinlock_profiling.cc:243] Waited 59.2 ms on lock 0x7b5400609d54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:08.211558 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136148224}
I20260202 15:09:08.213269 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.214318 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.160391 13362 raft_consensus.cc:1275] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260202 15:09:08.161298 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1698) took 2306 ms. Trace:
I20260202 15:09:08.222478 12720 raft_consensus.cc:2468] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:08.162081 12799 raft_consensus.cc:1240] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260202 15:09:08.146606 13326 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:08.224514 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 78021.2us
I20260202 15:09:08.223156 12719 rpcz_store.cc:276] 0202 15:09:05.854828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.864471 (+ 9643us) service_pool.cc:225] Handling call
0202 15:09:05.864710 (+ 239us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.218422 (+1353712us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.218455 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.218502 (+ 47us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:07.218539 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.161066 (+942527us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54000172d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:08.161120 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.161240 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3112800512}
I20260202 15:09:08.226884 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:08.146067 15647 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.000s sys 0.012s
I20260202 15:09:08.158322 12642 pending_rounds.cc:85] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 6
W20260202 15:09:08.174202 15716 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.003s sys 0.008s
I20260202 15:09:08.218235 15608 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.196274 12641 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:08.197824 15508 leader_election.cc:290] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.197785 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.189560 15743 raft_consensus.cc:3055] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:08.219964 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1707) took 1950 ms. Trace:
I20260202 15:09:08.261345 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1632) took 1356 ms. Trace:
I20260202 15:09:08.162496 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.232637 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 452) took 2990 ms. Trace:
I20260202 15:09:08.299403 12799 rpcz_store.cc:276] 0202 15:09:05.233375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.233523 (+ 148us) service_pool.cc:225] Handling call
0202 15:09:05.299043 (+ 65520us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:08.161805 (+2862762us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b54001f4120. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4d6df0 00007f24ed560e40 00007f24ed561eb1 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c
0202 15:09:08.223882 (+ 62077us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.224040 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9849728}
I20260202 15:09:08.233002 15446 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 10 FOLLOWER]: Leader election won for term 10
W20260202 15:09:08.240046 15647 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s user 0.000s sys 0.013s
I20260202 15:09:08.233992 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.309238 12720 raft_consensus.cc:2393] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:08.227458 13357 raft_consensus.cc:2468] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:08.134027 15540 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:08.319365 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:08.319941 12720 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:08.236521 12642 pending_rounds.cc:107] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 8.7
W20260202 15:09:08.177342 15694 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.009s sys 0.008s
I20260202 15:09:08.158634 15690 consensus_queue.cc:1059] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.037s
I20260202 15:09:08.160538 15756 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.244386 13356 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 9.
I20260202 15:09:08.328929 12800 raft_consensus.cc:2410] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260202 15:09:08.251114 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:09:08.330025 12650 raft_consensus.cc:606] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:08.251715 15588 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.254921 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.330955 12725 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 4.
I20260202 15:09:08.331456 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1712) took 1046 ms. Trace:
I20260202 15:09:08.331707 12725 rpcz_store.cc:276] 0202 15:09:07.284578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.226350 (+941772us) service_pool.cc:225] Handling call
0202 15:09:08.331398 (+105048us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.196146 15729 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:09:08.224670 15732 consensus_meta.cc:287] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.957s user 0.010s sys 0.000s
I20260202 15:09:08.184723 15671 consensus_queue.cc:260] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.283360 12641 raft_consensus.cc:1240] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:08.265280 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.290114 12645 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 9.
I20260202 15:09:08.333412 12723 raft_consensus.cc:2468] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:08.333799 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1708) took 1051 ms. Trace:
I20260202 15:09:08.334030 12645 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260202 15:09:08.334100 12723 rpcz_store.cc:276] 0202 15:09:07.282827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.264979 (+982152us) service_pool.cc:225] Handling call
0202 15:09:08.333754 (+ 68775us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.156824 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.259867 15718 leader_election.cc:290] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:08.305594 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.233496 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.341195 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1833) took 1068 ms. Trace:
I20260202 15:09:08.341629 12719 rpcz_store.cc:276] 0202 15:09:07.274166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.233302 (+959136us) service_pool.cc:225] Handling call
0202 15:09:08.341143 (+107841us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.342553 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.343364 12719 raft_consensus.cc:2393] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 9: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:08.295598 12724 rpcz_store.cc:276] 0202 15:09:06.269706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.289302 (+ 19596us) service_pool.cc:225] Handling call
0202 15:09:06.457712 (+168410us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.250549 (+792837us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.250577 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.250614 (+ 37us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:07.250647 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.219727 (+969080us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b540019a854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:08.219780 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.219903 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1822958336}
W20260202 15:09:08.151844 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 44391us
I20260202 15:09:08.239279 15720 raft_consensus.cc:3055] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:08.320935 12642 raft_consensus.cc:2889] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:08.239749 15674 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.326241 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.345598 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.346086 12724 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:09:08.346280 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.346778 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:08.326390 15694 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s user 0.009s sys 0.011s
I20260202 15:09:08.347723 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.348284 15671 raft_consensus.cc:3060] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:08.350180 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1631) took 1535 ms. Trace:
I20260202 15:09:08.350908 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.351677 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1704) took 2090 ms. Trace:
I20260202 15:09:08.351835 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.352463 12725 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 8: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:08.352005 12717 rpcz_store.cc:276] 0202 15:09:06.261608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.261811 (+ 203us) service_pool.cc:225] Handling call
0202 15:09:06.266025 (+ 4214us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.626876 (+1360851us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:09:07.626905 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:07.626946 (+ 41us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:07.626986 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.351510 (+724524us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400145854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:08.351545 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.351631 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":121823,"raft.queue_time_us":74561,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":81633,"spinlock_wait_cycles":2958766464}
I20260202 15:09:08.326306 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.353245 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.353797 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.354056 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.354727 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:08.258627 15414 consensus_meta.cc:287] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.785s user 0.002s sys 0.008s
I20260202 15:09:08.355639 15712 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.356060 15414 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:08.356637 13363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 891) took 3120 ms (client timeout 3479 ms). Trace:
W20260202 15:09:08.356953 13363 rpcz_store.cc:269] 0202 15:09:05.236703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.236922 (+ 219us) service_pool.cc:225] Handling call
0202 15:09:08.356487 (+3119565us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b5400650c54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:08.356589 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5602008320}
I20260202 15:09:08.357745 15414 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.357828 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:08.264264 15721 consensus_meta.cc:287] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.004s user 0.006s sys 0.003s
I20260202 15:09:08.357985 15712 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.358414 15721 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.360268 15721 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:08.348307 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1717) took 1051 ms. Trace:
I20260202 15:09:08.363081 12724 rpcz_store.cc:276] 0202 15:09:07.296787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.347526 (+1050739us) service_pool.cc:225] Handling call
0202 15:09:08.348261 (+ 735us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:09:08.251770 15762 log.cc:927] Time spent T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260202 15:09:08.202855 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1702) took 1945 ms. Trace:
I20260202 15:09:08.263332 15695 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:08.262924 15626 raft_consensus.cc:3055] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:08.301370 12797 rpcz_store.cc:276] 0202 15:09:06.904975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.271849 (+366874us) service_pool.cc:225] Handling call
0202 15:09:07.272090 (+ 241us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.087387 (+815297us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.261115 (+173728us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b5400357ad4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:08.261297 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":610063104}
I20260202 15:09:08.336261 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85ea3f0239bb4603a3a363108154ef77" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.366420 13357 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20260202 15:09:08.297405 15743 raft_consensus.cc:740] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:08.370334 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:08.344911 12791 raft_consensus.cc:2393] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:08.340045 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.351014 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 726) took 3012 ms. Trace:
I20260202 15:09:08.371644 13360 raft_consensus.cc:2468] T f72d171c95a149d890432428ecf80739 P 66b8c715b9e6479e9218b08bd3156380 [term 8 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8.
I20260202 15:09:08.371588 13354 rpcz_store.cc:276] 0202 15:09:05.338338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.339220 (+ 882us) service_pool.cc:225] Handling call
0202 15:09:05.861143 (+521923us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:05.896089 (+ 34946us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.350860 (+2454771us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b540059e2d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:08.350968 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":678036352}
I20260202 15:09:08.350477 12798 rpcz_store.cc:276] 0202 15:09:06.815002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.271765 (+456763us) service_pool.cc:225] Handling call
0202 15:09:07.271994 (+ 229us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.103981 (+831987us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.350000 (+246019us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b54001cd4d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:08.350133 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":861837568}
I20260202 15:09:08.358379 13363 raft_consensus.cc:2468] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8.
I20260202 15:09:08.370334 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.345317 12688 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.365303 12646 consensus_queue.cc:237] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.355655 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.367076 13358 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44934 (request call id 717) took 1019 ms. Trace:
I20260202 15:09:08.285090 15673 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:08.375186 13358 rpcz_store.cc:276] 0202 15:09:07.347254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.309066 (+961812us) service_pool.cc:225] Handling call
0202 15:09:08.367028 (+ 57962us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.367122 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:08.326946 15446 raft_consensus.cc:697] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 10 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:08.377645 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:08.367620 12720 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8.
I20260202 15:09:08.378520 12797 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:08.379230 12720 raft_consensus.cc:1240] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260202 15:09:08.379998 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 444) took 1115 ms. Trace:
I20260202 15:09:08.380247 12720 rpcz_store.cc:276] 0202 15:09:07.264934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.378718 (+1113784us) service_pool.cc:225] Handling call
0202 15:09:08.379022 (+ 304us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:08.379717 (+ 695us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.379934 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.381340 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1713) took 1096 ms. Trace:
I20260202 15:09:08.381573 12720 rpcz_store.cc:276] 0202 15:09:07.284821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.380760 (+1095939us) service_pool.cc:225] Handling call
0202 15:09:08.380976 (+ 216us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.381110 (+ 134us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:08.381130 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:08.381161 (+ 31us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:08.381188 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.381228 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.381304 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.352785 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1710) took 1069 ms. Trace:
I20260202 15:09:08.351908 15756 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:08.365840 15720 raft_consensus.cc:740] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:08.385635 15720 consensus_queue.cc:260] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.386418 15743 consensus_queue.cc:260] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.387813 15743 raft_consensus.cc:3060] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:08.363860 12722 rpcz_store.cc:276] 0202 15:09:06.257707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.257809 (+ 102us) service_pool.cc:225] Handling call
0202 15:09:06.258135 (+ 326us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:06.259679 (+ 1544us) raft_consensus.cc:1534] Early marking committed up to index 7
0202 15:09:06.259710 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:06.266407 (+ 6697us) log.cc:844] Serialized 26 byte log entry
0202 15:09:07.494435 (+1228028us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b18001b52c0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471 00007f24ed51a73f
0202 15:09:08.129816 (+635381us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:09:08.129872 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.129937 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:08.202597 (+ 72660us) raft_consensus.cc:1685] finished
0202 15:09:08.202616 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.202806 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":242099,"raft.queue_time_us":2714,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":10644,"spinlock_wait_cycles":2874240,"thread_start_us":635187,"threads_started":1,"wal-append.queue_time_us":1908577}
I20260202 15:09:08.369273 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.336102 15729 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:08.391793 12722 raft_consensus.cc:1240] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:08.392449 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1716) took 1100 ms. Trace:
I20260202 15:09:08.392771 12722 rpcz_store.cc:276] 0202 15:09:07.292518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.391314 (+1098796us) service_pool.cc:225] Handling call
0202 15:09:08.391599 (+ 285us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.392209 (+ 610us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.392395 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.375416 12799 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:09:08.376130 12688 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.376498 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:08.329376 15540 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.378456 12650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "30df58c1134a49e18c29202abd31da41"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:09:08.339103 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.382887 12725 rpcz_store.cc:276] 0202 15:09:07.283311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.351631 (+1068320us) service_pool.cc:225] Handling call
0202 15:09:08.352743 (+ 1112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.399812 12719 raft_consensus.cc:1240] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:08.384377 12720 raft_consensus.cc:1240] T f72d171c95a149d890432428ecf80739 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260202 15:09:08.398656 12724 raft_consensus.cc:2410] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260202 15:09:08.368268 15647 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:08.329715 15574 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.403332 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 445) took 1138 ms. Trace:
I20260202 15:09:08.403757 12724 rpcz_store.cc:276] 0202 15:09:07.271489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.363642 (+1092153us) service_pool.cc:225] Handling call
0202 15:09:08.403280 (+ 39638us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.196282 15726 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:08.401435 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.401657 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1714) took 1116 ms. Trace:
I20260202 15:09:08.405500 12720 rpcz_store.cc:276] 0202 15:09:07.285107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.382230 (+1097123us) service_pool.cc:225] Handling call
0202 15:09:08.384013 (+ 1783us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:08.401343 (+ 17330us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.401614 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.369913 15626 raft_consensus.cc:740] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:08.406822 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40256 (request call id 8) took 1036 ms. Trace:
I20260202 15:09:08.412524 13356 rpcz_store.cc:276] 0202 15:09:07.366072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.371105 (+1005033us) service_pool.cc:225] Handling call
0202 15:09:08.402163 (+ 31058us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.403213 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.415027 15626 consensus_queue.cc:260] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.416344 15626 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:08.403442 15647 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:08.417488 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 442) took 1153 ms. Trace:
I20260202 15:09:08.417757 15647 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:08.417853 12726 rpcz_store.cc:276] 0202 15:09:07.264436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.264545 (+ 109us) service_pool.cc:225] Handling call
0202 15:09:07.264815 (+ 270us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:07.265147 (+ 332us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.265174 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:07.265668 (+ 494us) log.cc:844] Serialized 52 byte log entry
0202 15:09:07.492571 (+226903us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:07.492608 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:07.492667 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:08.417274 (+924607us) raft_consensus.cc:1685] finished
0202 15:09:08.417295 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.417442 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2637,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"thread_start_us":226510,"threads_started":1,"wal-append.queue_time_us":812438}
I20260202 15:09:08.339540 15732 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.420392 13361 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20260202 15:09:08.421264 15732 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:08.422042 15733 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:08.422501 15733 raft_consensus.cc:493] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:08.402392 12643 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:08.408643 15608 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:08.422917 15733 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:08.410123 12797 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 9.
I20260202 15:09:08.428220 15540 leader_election.cc:290] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.428463 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1751) took 1098 ms. Trace:
I20260202 15:09:08.428967 12643 rpcz_store.cc:276] 0202 15:09:07.329972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.383018 (+ 53046us) service_pool.cc:225] Handling call
0202 15:09:08.428387 (+1045369us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.430568 15574 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:08.432006 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 456) took 1071 ms. Trace:
I20260202 15:09:08.432377 12795 rpcz_store.cc:276] 0202 15:09:07.360987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.496520 (+135533us) service_pool.cc:225] Handling call
0202 15:09:08.431802 (+935282us) spinlock_profiling.cc:243] Waited 51 ms on lock 0x7b540004c254. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:08.431935 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117352832}
I20260202 15:09:08.434041 12647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1738) took 3225 ms. Trace:
W20260202 15:09:08.435822 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.434427 12647 rpcz_store.cc:276] 0202 15:09:05.208480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.256754 (+2048274us) service_pool.cc:225] Handling call
0202 15:09:07.453801 (+197047us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:07.454110 (+ 309us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:07.454136 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:07.454507 (+ 371us) log.cc:844] Serialized 26 byte log entry
0202 15:09:08.097363 (+642856us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:09:08.097394 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.097448 (+ 54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:08.286957 (+189509us) raft_consensus.cc:1685] finished
0202 15:09:08.286982 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.433787 (+146805us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54002a9750. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:08.433979 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":907,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":71,"spinlock_wait_cycles":452547328,"thread_start_us":642599,"threads_started":1,"wal-append.queue_time_us":765225}
I20260202 15:09:08.396914 15712 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:08.445590 12688 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:08.444578 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 327153us
I20260202 15:09:08.436211 15726 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:08.378006 15446 consensus_queue.cc:237] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.447113 12688 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.444643 15712 raft_consensus.cc:493] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:08.466331 12688 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.448071 15736 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:08.448127 15414 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:08.487138 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:08.487738 12643 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:08.471462 15608 raft_consensus.cc:2804] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:08.494531 13361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 740) took 2424 ms. Trace:
I20260202 15:09:08.497256 13361 rpcz_store.cc:276] 0202 15:09:06.069787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.252840 (+1183053us) service_pool.cc:225] Handling call
0202 15:09:07.253177 (+ 337us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:08.420807 (+1167630us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.494331 (+ 73524us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400264b54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:08.494482 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2683570944}
I20260202 15:09:08.493335 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:08.498427 12646 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:08.495939 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:08.499468 12645 raft_consensus.cc:2410] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260202 15:09:08.505334 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.505649 12643 raft_consensus.cc:2468] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:08.506588 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:08.465953 15732 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:08.482646 15712 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:08.521281 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:08.521585 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:08.521966 13361 raft_consensus.cc:2410] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260202 15:09:08.523638 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.525396 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.530705 12688 leader_election.cc:304] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.508256 15414 raft_consensus.cc:2804] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader election won for term 9
W20260202 15:09:08.538198 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.522053 13358 raft_consensus.cc:2376] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8. Re-sending same reply.
W20260202 15:09:08.511520 15417 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.007s sys 0.022s
I20260202 15:09:08.496834 15608 raft_consensus.cc:493] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:08.544796 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.550163 13356 raft_consensus.cc:1275] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260202 15:09:08.552593 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.552796 13356 pending_rounds.cc:85] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: Aborting all ops after (but not including) 6
I20260202 15:09:08.553092 13356 pending_rounds.cc:107] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260202 15:09:08.553339 13356 raft_consensus.cc:2889] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:08.554234 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.554891 13361 raft_consensus.cc:2393] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 9: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:08.559043 12646 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:08.565704 13360 raft_consensus.cc:2393] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 7: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:08.566069 13363 raft_consensus.cc:2393] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 9: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:08.559542 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:08.570179 12688 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:08.570420 12765 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:08.571689 12765 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:08.572140 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:08.550239 15417 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.018s user 0.008s sys 0.023s
I20260202 15:09:08.576501 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:08.577662 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:08.578492 12723 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:09:08.573328 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.568848 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:08.573041 12765 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:09:08.580271 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 448524us
W20260202 15:09:08.575179 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 475982us
I20260202 15:09:08.582873 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:08.565804 15608 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:08.586267 12800 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:08.576088 15588 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:09:08.590322 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 451623us
I20260202 15:09:08.567308 15414 raft_consensus.cc:697] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:09:08.592854 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 466214us
I20260202 15:09:08.517081 15732 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.545439 15569 consensus_queue.cc:1059] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260202 15:09:08.569022 15572 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:08.568857 15721 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:08.600131 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.601080 12796 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32824 (request call id 453) took 1240 ms. Trace:
I20260202 15:09:08.601447 12796 rpcz_store.cc:276] 0202 15:09:07.360202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.592645 (+1232443us) service_pool.cc:225] Handling call
0202 15:09:08.601012 (+ 8367us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:08.602102 12646 raft_consensus.cc:1275] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260202 15:09:08.591181 12614 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:08.580190 15503 raft_consensus.cc:2804] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election won for term 6
W20260202 15:09:08.597438 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:08.603390 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.589921 15588 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:08.604696 15588 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:08.606859 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1700) took 2352 ms. Trace:
I20260202 15:09:08.594588 15729 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.612447 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:08.615856 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:08.617089 12646 raft_consensus.cc:1275] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260202 15:09:08.622258 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:08.622876 13360 raft_consensus.cc:3060] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:09:08.574486 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 477042us
I20260202 15:09:08.595341 15626 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:08.645897 12717 raft_consensus.cc:2393] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:08.647658 15776 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:08.650403 13360 raft_consensus.cc:2468] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
W20260202 15:09:08.652038 12792 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.004s sys 0.020s
W20260202 15:09:08.652381 12764 connection.cc:232] RPC call timeout handler was delayed by -2.0828006830001868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19590000000000002s for a response
W20260202 15:09:08.653087 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:08.664271 12792 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s user 0.006s sys 0.020s
W20260202 15:09:08.656003 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:08.663265 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.652632 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:08.665125 12796 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:08.621596 15780 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260202 15:09:08.666307 12764 connection.cc:232] RPC call timeout handler was delayed by -1.6567685669999719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2657s for a response
W20260202 15:09:08.665993 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 43925.4us
I20260202 15:09:08.672571 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:08.673506 12764 connection.cc:232] RPC call timeout handler was delayed by -1.1264849010001399s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18080000000000002s for a response
I20260202 15:09:08.673847 13354 raft_consensus.cc:2393] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:08.609651 15721 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:08.676880 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 103907us
W20260202 15:09:08.676816 13328 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.027s sys 0.015s
W20260202 15:09:08.677976 13328 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.028s sys 0.015s
I20260202 15:09:08.678053 12765 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:08.681432 12763 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.682206 12763 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:09:08.682938 12532 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 9fae829b1c574bc795c5a424036329b3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 66b8c715b9e6479e9218b08bd3156380 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:09:08.684183 15574 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:08.684549 15574 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:08.684870 15574 raft_consensus.cc:3060] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:08.686204 15706 raft_consensus.cc:2804] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:08.686640 15706 raft_consensus.cc:697] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:08.665856 15503 raft_consensus.cc:697] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:08.688398 12765 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:08.621448 15572 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.699095 12726 raft_consensus.cc:1275] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260202 15:09:08.699481 12726 pending_rounds.cc:85] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Aborting all ops after (but not including) 3
I20260202 15:09:08.699725 12726 pending_rounds.cc:107] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260202 15:09:08.699946 12726 raft_consensus.cc:2889] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:08.663185 15626 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.701561 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32824 (request call id 449) took 3510 ms. Trace:
I20260202 15:09:08.679484 15508 raft_consensus.cc:2804] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260202 15:09:08.704155 12792 rpcz_store.cc:276] 0202 15:09:05.190783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:05.206202 (+ 15419us) service_pool.cc:225] Handling call
0202 15:09:05.230400 (+ 24198us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:05.302117 (+ 71717us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:05.302145 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:06.624897 (+1322752us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b180040d4a0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea47f2d6 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ebe5d9b1 00007f24ebe6cfe2 00007f24ed59a851 00007f24ed55f0ba
0202 15:09:06.625400 (+ 503us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b2c0003d748. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ebe5d9b1 00007f24ebe6cfe2 00007f24ed59a851 00007f24ed55f0ba 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22
0202 15:09:06.626001 (+ 601us) log.cc:844] Serialized 52 byte log entry
0202 15:09:07.437256 (+811255us) raft_consensus.cc:1645] Marking committed up to 7
0202 15:09:08.100588 (+663332us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.665736 (+565148us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.688765 (+ 23029us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.693276 (+ 4511us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:08.701387 (+ 8111us) raft_consensus.cc:1685] finished
0202 15:09:08.701406 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.701515 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45944,"prepare.queue_time_us":1240231,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":120138,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":80899,"spinlock_wait_cycles":31705088,"thread_start_us":1938055,"threads_started":3,"wal-append.queue_time_us":840583}
W20260202 15:09:08.688279 15771 log.cc:927] Time spent T ebc29e67ee294975abcdfc9e53f8b6e1 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260202 15:09:08.703320 13355 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.007s sys 0.017s
W20260202 15:09:08.711261 13355 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s user 0.007s sys 0.017s
I20260202 15:09:08.607291 12718 rpcz_store.cc:276] 0202 15:09:06.254431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.254645 (+ 214us) service_pool.cc:225] Handling call
0202 15:09:06.255028 (+ 383us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:06.263696 (+ 8668us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:06.263741 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:06.268697 (+ 4956us) log.cc:844] Serialized 52 byte log entry
0202 15:09:07.196210 (+927513us) raft_consensus.cc:1645] Marking committed up to 4
0202 15:09:08.135656 (+939446us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.606534 (+470878us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.606618 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:08.606655 (+ 37us) raft_consensus.cc:1685] finished
0202 15:09:08.606665 (+ 10us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.606798 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29880,"prepare.queue_time_us":1245302,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":88272,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":17328,"spinlock_wait_cycles":1280,"thread_start_us":1624461,"threads_started":3,"wal-append.queue_time_us":1172000}
I20260202 15:09:08.710079 15508 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:08.715365 15508 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:08.721410 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44942 (request call id 742) took 2569 ms. Trace:
I20260202 15:09:08.727231 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1705) took 2460 ms. Trace:
I20260202 15:09:08.700451 15735 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:08.724232 13355 rpcz_store.cc:276] 0202 15:09:06.152360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.255144 (+1102784us) service_pool.cc:225] Handling call
0202 15:09:07.255371 (+ 227us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.255538 (+ 167us) raft_consensus.cc:1534] Early marking committed up to index 8
0202 15:09:08.130465 (+874927us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.712503 (+582038us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.721007 (+ 8504us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:08.721068 (+ 61us) raft_consensus.cc:1645] Marking committed up to 8
0202 15:09:08.721126 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.721209 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.721342 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8139,"prepare.queue_time_us":999275,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":10974,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":145,"spinlock_wait_cycles":760704,"thread_start_us":1455550,"threads_started":2}
I20260202 15:09:08.727682 12721 rpcz_store.cc:276] 0202 15:09:06.266653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:06.266810 (+ 157us) service_pool.cc:225] Handling call
0202 15:09:06.276917 (+ 10107us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:06.489213 (+212296us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:06.489246 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:06.489292 (+ 46us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:09:06.489327 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.726955 (+2237628us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54003ac0d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:08.727009 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.727165 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":487763840}
I20260202 15:09:08.698361 15676 raft_consensus.cc:493] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:08.648088 15776 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:08.746511 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.742604 13328 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:08.703217 15743 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260202 15:09:08.700537 15784 raft_consensus.cc:2804] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
W20260202 15:09:08.748610 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 134395us
W20260202 15:09:08.755702 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 72236.5us
W20260202 15:09:08.756413 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 186266us
I20260202 15:09:08.757494 12614 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.760255 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:08.760838 12646 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:09:08.751447 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 154113us
I20260202 15:09:08.761891 12688 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.763226 12689 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:08.687373 15706 consensus_queue.cc:237] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.745344 15735 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:08.688597 15414 consensus_queue.cc:237] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.703395 15780 consensus_queue.cc:1059] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260202 15:09:08.754583 15775 raft_consensus.cc:2804] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:08.785172 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:08.787793 12795 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:08.786375 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:08.789111 12797 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:08.798729 12650 raft_consensus.cc:606] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:08.799216 12650 raft_consensus.cc:614] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260202 15:09:08.802337 12797 raft_consensus.cc:2468] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 9.
I20260202 15:09:08.747491 15778 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260202 15:09:08.805474 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.806758 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.782979 15735 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:08.791064 15775 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:08.805804 12646 raft_consensus.cc:2468] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:08.782971 15674 raft_consensus.cc:493] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:08.803984 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:08.768072 15784 raft_consensus.cc:697] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:09:08.787130 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 113198us
I20260202 15:09:08.800066 15729 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:08.754357 15676 raft_consensus.cc:515] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.811631 13328 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:08.767935 15791 consensus_queue.cc:1048] T 015f643434d84eac8a3774b663eb4220 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.054s
I20260202 15:09:08.811636 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:08.805601 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 105419us
I20260202 15:09:08.815371 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:08.826998 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1714) took 1400 ms. Trace:
I20260202 15:09:08.815372 12688 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.820468 12798 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:09:08.822260 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:08.827677 12791 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:08.827770 12646 raft_consensus.cc:2393] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 8: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
W20260202 15:09:08.816181 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 58748.7us
I20260202 15:09:08.833340 12795 raft_consensus.cc:2468] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:09:08.783882 15503 consensus_queue.cc:237] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.825014 15569 raft_consensus.cc:2804] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:09:08.836910 12648 rpcz_store.cc:276] 0202 15:09:07.427094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.197821 (+770727us) service_pool.cc:225] Handling call
0202 15:09:08.198041 (+ 220us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.198204 (+ 163us) raft_consensus.cc:1534] Early marking committed up to index 6
0202 15:09:08.241847 (+ 43643us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.798467 (+556620us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:08.798532 (+ 65us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:08.798589 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.826852 (+ 28263us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.826951 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":43504,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":165664,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":2626,"thread_start_us":556221,"threads_started":2,"wal-append.queue_time_us":497464}
I20260202 15:09:08.819772 15775 raft_consensus.cc:3060] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:08.844902 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:08.824273 15743 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260202 15:09:08.828128 15671 raft_consensus.cc:2804] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader election won for term 9
I20260202 15:09:08.847827 12643 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 9.
I20260202 15:09:08.814728 15783 consensus_queue.cc:1048] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260202 15:09:08.847961 12614 leader_election.cc:304] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:08.821807 15733 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.843509 15673 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:08.809535 15805 consensus_queue.cc:1048] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260202 15:09:08.844987 15721 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:08.856848 12688 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.846864 15810 raft_consensus.cc:493] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:08.863107 13357 raft_consensus.cc:1240] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 10 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260202 15:09:08.846875 15801 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:08.823331 15674 raft_consensus.cc:515] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.856753 12791 raft_consensus.cc:2468] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
I20260202 15:09:08.857928 12614 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.868678 15673 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.852891 15569 raft_consensus.cc:697] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [term 8 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:08.852329 15732 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:08.824651 15797 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
W20260202 15:09:08.902802 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 45217.2us
W20260202 15:09:08.906466 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 50802.2us
I20260202 15:09:08.931157 12614 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:09:08.931471 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:08.863613 15671 raft_consensus.cc:697] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [term 9 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:08.913829 15791 raft_consensus.cc:3055] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Stepping down as leader of term 8
I20260202 15:09:08.913699 15508 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.898628 15784 consensus_queue.cc:237] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:08.918432 15773 log.cc:927] Time spent T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
I20260202 15:09:08.856933 15626 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.857981 15662 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:08.913069 15572 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:08.933200 12614 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:08.857869 15804 raft_consensus.cc:2804] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:09:08.906169 15786 log.cc:927] Time spent T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
I20260202 15:09:08.898475 15799 raft_consensus.cc:2804] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:08.940268 15791 raft_consensus.cc:740] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:08.970707 12796 raft_consensus.cc:2410] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260202 15:09:08.973063 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1649) took 1171 ms. Trace:
W20260202 15:09:08.958673 15788 log.cc:927] Time spent T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
W20260202 15:09:08.986593 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 53450.3us
W20260202 15:09:08.987953 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 358784us
I20260202 15:09:08.975162 12799 rpcz_store.cc:276] 0202 15:09:07.802681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.395685 (+593004us) service_pool.cc:225] Handling call
0202 15:09:08.435680 (+ 39995us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.435921 (+ 241us) raft_consensus.cc:1534] Early marking committed up to index 3
0202 15:09:08.667491 (+231570us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.971198 (+303707us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.971461 (+ 263us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:08.971663 (+ 202us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:08.971844 (+ 181us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.972440 (+ 596us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:08.972913 (+ 473us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19581,"prepare.queue_time_us":272761,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":1432,"raft.queue_time_us":51840,"raft.run_wall_time_us":11311,"spinlock_wait_cycles":2519424,"thread_start_us":486712,"threads_started":3,"wal-append.queue_time_us":172565}
I20260202 15:09:08.944097 15776 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:08.905562 15588 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.956199 15729 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260202 15:09:08.921280 15801 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:08.967890 15780 raft_consensus.cc:2749] T 4ed5f5e04d064d6e8d431aa2209cc141 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:08.962090 15804 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
W20260202 15:09:09.001099 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 170518us
I20260202 15:09:08.972244 15417 raft_consensus.cc:2749] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:08.905272 15810 raft_consensus.cc:515] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:09.004182 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 182178us
I20260202 15:09:08.985473 15756 raft_consensus.cc:493] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:08.988466 15799 raft_consensus.cc:697] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:08.996335 15729 raft_consensus.cc:493] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:08.972730 15572 consensus_queue.cc:1059] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260202 15:09:08.961676 15662 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:08.994432 15789 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:08.967582 15813 consensus_queue.cc:1059] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
W20260202 15:09:09.040499 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 49989us
I20260202 15:09:09.014494 15729 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Advancing to term 10
I20260202 15:09:09.001181 15792 raft_consensus.cc:2804] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:09.026356 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1651) took 1016 ms. Trace:
I20260202 15:09:09.007440 15804 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:09.009194 15800 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260202 15:09:09.043702 12800 rpcz_store.cc:276] 0202 15:09:08.010199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.410452 (+400253us) service_pool.cc:225] Handling call
0202 15:09:08.410739 (+ 287us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.589023 (+178284us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:09.026161 (+437138us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54003728d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:09.026306 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":403169536}
I20260202 15:09:08.997251 15806 consensus_queue.cc:1059] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
W20260202 15:09:09.046352 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 223854us
I20260202 15:09:09.047219 12764 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:09.049003 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:09.048568 12690 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:09:09.052917 15698 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.564s user 0.015s sys 0.014s
I20260202 15:09:09.053618 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.054143 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:08.993796 15569 consensus_queue.cc:237] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:09.048943 15796 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20260202 15:09:09.058673 13328 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:09.049880 15800 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:09.045159 15792 raft_consensus.cc:493] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:09:09.053418 15698 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.565s user 0.015s sys 0.014s
I20260202 15:09:09.018342 15756 raft_consensus.cc:515] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.079874 13328 leader_election.cc:304] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:09.019016 15814 raft_consensus.cc:2804] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260202 15:09:09.053889 15795 log.cc:927] Time spent T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20260202 15:09:09.009150 15671 consensus_queue.cc:237] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.048060 15414 consensus_queue.cc:1048] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260202 15:09:09.085916 15673 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:09.085716 15799 consensus_queue.cc:237] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:09.044703 15743 consensus_queue.cc:1048] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260202 15:09:09.086738 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1850) took 1629 ms. Trace:
I20260202 15:09:09.089277 15446 raft_consensus.cc:2804] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:09.090517 12643 raft_consensus.cc:1240] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 6. Current term is 7. Ops: [5.4-6.5]
W20260202 15:09:09.090950 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 93184.9us
W20260202 15:09:09.059069 15790 log.cc:927] Time spent T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20260202 15:09:09.042265 15735 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.091655 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 894) took 1875 ms. Trace:
I20260202 15:09:09.143893 13359 rpcz_store.cc:276] 0202 15:09:07.215870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.265200 (+ 49330us) service_pool.cc:225] Handling call
0202 15:09:07.534503 (+269303us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.085588 (+551085us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:09.091479 (+1005891us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54005d2d50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:09.091596 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":618733824}
W20260202 15:09:09.094182 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 47082.7us
I20260202 15:09:09.145677 12764 leader_election.cc:304] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:09.099653 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.069957 15732 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:09.054801 15813 consensus_queue.cc:1059] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260202 15:09:09.050964 15783 consensus_queue.cc:1048] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260202 15:09:09.080749 15733 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:09.137820 12717 rpcz_store.cc:276] 0202 15:09:07.457381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.353884 (+896503us) service_pool.cc:225] Handling call
0202 15:09:09.086514 (+732630us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b54002fe754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:09.086659 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":669455104}
I20260202 15:09:09.087124 15662 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:09.138366 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 116612us
W20260202 15:09:09.095631 15803 log.cc:927] Time spent T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.224s user 0.001s sys 0.001s
I20260202 15:09:09.082296 15800 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Advancing to term 10
I20260202 15:09:09.142146 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:09.164716 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.089666 15792 raft_consensus.cc:3060] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:09.020310 15791 consensus_queue.cc:260] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.097110 15774 raft_consensus.cc:2804] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:09:09.152043 12764 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:09.091754 15814 raft_consensus.cc:493] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:09:09.138567 15809 log.cc:927] Time spent T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.217s user 0.000s sys 0.000s
I20260202 15:09:09.157886 12765 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:09.079128 15572 consensus_queue.cc:1048] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260202 15:09:09.156950 15446 raft_consensus.cc:697] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:09.159518 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.161387 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.049409 15789 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.168666 12792 raft_consensus.cc:1240] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:09.169029 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.138446 15825 raft_consensus.cc:2804] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:09.148365 15732 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:09:09.204913 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 52956.5us
I20260202 15:09:09.171589 12649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34792 (request call id 1708) took 2804 ms. Trace:
W20260202 15:09:09.178174 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.091560 15817 consensus_queue.cc:1048] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.083s
W20260202 15:09:09.158215 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 390473us
I20260202 15:09:09.171732 15756 leader_election.cc:290] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:09:09.202648 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.204532 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.165757 15508 leader_election.cc:290] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:09.168668 15806 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260202 15:09:09.210137 12764 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:09.210168 12649 rpcz_store.cc:276] 0202 15:09:06.367032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.179803 (+1812771us) service_pool.cc:225] Handling call
0202 15:09:08.228336 (+ 48533us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:08.228698 (+ 362us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:08.228737 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:08.229218 (+ 481us) log.cc:844] Serialized 26 byte log entry
0202 15:09:08.663983 (+434765us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:08.667440 (+ 3457us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.667770 (+ 330us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:09.171414 (+503644us) raft_consensus.cc:1685] finished
0202 15:09:09.171432 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:09.171539 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":200131,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":375765,"threads_started":1,"wal-append.queue_time_us":517727}
I20260202 15:09:09.215963 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:09.271946 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 61893.5us
I20260202 15:09:09.272694 12764 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:09.273860 12764 leader_election.cc:304] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:09.216202 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:09:09.189781 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.218010 15815 raft_consensus.cc:3055] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:09.219213 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:09:09.221278 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 63509.1us
I20260202 15:09:09.199452 15588 leader_election.cc:290] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:09.198946 15814 raft_consensus.cc:3060] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:09.223414 15824 raft_consensus.cc:2749] T 85ea3f0239bb4603a3a363108154ef77 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260202 15:09:09.225648 15726 consensus_meta.cc:287] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.759s user 0.009s sys 0.000s
I20260202 15:09:09.225797 15829 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260202 15:09:09.295876 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:09.226276 15834 raft_consensus.cc:2749] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:09.230011 15743 consensus_queue.cc:1059] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:09:09.215725 15732 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:09.230875 15414 consensus_queue.cc:1048] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260202 15:09:09.233093 15712 consensus_meta.cc:287] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.697s user 0.009s sys 0.000s
I20260202 15:09:09.232933 13328 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:09.216391 15647 consensus_meta.cc:287] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.798s user 0.006s sys 0.006s
W20260202 15:09:09.237071 15574 consensus_meta.cc:287] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.552s user 0.009s sys 0.000s
I20260202 15:09:09.238361 15446 consensus_queue.cc:237] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.206403 15706 raft_consensus.cc:2804] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:09.344539 15706 raft_consensus.cc:493] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:09.344944 15706 raft_consensus.cc:3060] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:09.347069 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:09.241348 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:09.241761 15736 consensus_meta.cc:287] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.756s user 0.010s sys 0.000s
I20260202 15:09:09.207779 15825 raft_consensus.cc:493] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:09.265487 12647 raft_consensus.cc:1240] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:09.275712 12764 leader_election.cc:304] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:09.355340 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 79679.2us
I20260202 15:09:09.356532 12764 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:09.276788 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:09.277551 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:09.162984 15626 consensus_queue.cc:1048] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260202 15:09:09.278893 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:09.360502 13361 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:09.221167 15662 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:09.280913 12765 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:09.287453 12641 raft_consensus.cc:1240] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:09.290391 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d91ac50607b34cd1b7d61cd42f7c26d8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:09.291435 12649 raft_consensus.cc:1240] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: []
W20260202 15:09:09.275983 15793 log.cc:927] Time spent T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.257s user 0.000s sys 0.001s
I20260202 15:09:09.314443 13354 raft_consensus.cc:3060] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:09.299960 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:09.275087 15508 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:09.329689 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.330387 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.326905 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42782 (request call id 508) took 3143 ms. Trace:
I20260202 15:09:09.392189 13354 raft_consensus.cc:2468] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 9.
I20260202 15:09:09.428887 12796 raft_consensus.cc:2410] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260202 15:09:09.428608 12642 rpcz_store.cc:276] 0202 15:09:06.183053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.157744 (+1974691us) service_pool.cc:225] Handling call
0202 15:09:08.158099 (+ 355us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:08.711604 (+553505us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:08.711688 (+ 84us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:08.712682 (+ 994us) log.cc:844] Serialized 52 byte log entry
0202 15:09:08.801535 (+ 88853us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:08.801589 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.801664 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:09.326737 (+525073us) raft_consensus.cc:1685] finished
0202 15:09:09.326757 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:09.326864 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":83480,"raft.run_cpu_time_us":45,"raft.run_wall_time_us":45,"thread_start_us":411488,"threads_started":2,"wal-append.queue_time_us":127662}
I20260202 15:09:09.430559 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
W20260202 15:09:09.331601 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.391258 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:09.331375 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 98522.2us
I20260202 15:09:09.280233 15815 raft_consensus.cc:740] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:09.460501 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.383544 15838 raft_consensus.cc:2749] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:09.461781 15721 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:09.462064 15838 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260202 15:09:09.462152 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:09.462762 12647 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 5
I20260202 15:09:09.463560 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.332835 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.333132 13362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 716) took 2036 ms. Trace:
I20260202 15:09:09.465848 12799 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:09:09.465764 13362 rpcz_store.cc:276] 0202 15:09:07.296294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.299697 (+ 3403us) service_pool.cc:225] Handling call
0202 15:09:08.160053 (+860356us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.222953 (+ 62900us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:09.326187 (+1103234us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54006090d0. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:09.333079 (+ 6892us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1978129280}
I20260202 15:09:09.295519 15822 raft_consensus.cc:2749] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:09.334405 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:09.337632 15832 log.cc:927] Time spent T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260202 15:09:09.339202 15743 consensus_queue.cc:1243] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260202 15:09:09.342160 15826 log.cc:927] Time spent T af412f1b23854fdea68917d1df084a35 P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
W20260202 15:09:09.342657 15823 log.cc:927] Time spent T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.172s user 0.000s sys 0.001s
I20260202 15:09:09.334939 15712 raft_consensus.cc:515] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:09.343428 15725 raft_consensus.cc:3055] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Stepping down as leader of term 4
I20260202 15:09:09.343796 15574 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:09.300442 15801 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:09.353444 12793 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:09.353513 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.353979 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.357893 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:09.354097 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:09.382673 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.530673 12720 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 9.
I20260202 15:09:09.360014 15825 raft_consensus.cc:3060] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:09.336074 15647 raft_consensus.cc:515] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.571318 13362 raft_consensus.cc:1240] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: [3.5-4.6]
I20260202 15:09:09.383621 13359 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Advancing to term 9
I20260202 15:09:09.386163 12721 raft_consensus.cc:1240] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:09.390851 15662 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:09.390872 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9793f4e4e46418e44fbdc3d3a4794" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:09.397914 15789 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:09.391448 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.580994 15813 consensus_queue.cc:1243] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:09:09.581750 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.581913 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.583120 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.391721 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:09.361212 15820 log.cc:927] Time spent T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.271s user 0.000s sys 0.001s
I20260202 15:09:09.585058 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:09.405441 15775 consensus_meta.cc:287] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.551s user 0.005s sys 0.005s
I20260202 15:09:09.391916 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.411067 15810 leader_election.cc:290] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:09.411952 15735 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:09.335419 15835 raft_consensus.cc:2749] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:09.315107 15726 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:09.343472 15798 log.cc:927] Time spent T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
W20260202 15:09:09.416015 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 59557us
W20260202 15:09:09.416752 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 135862us
I20260202 15:09:09.464141 15815 consensus_queue.cc:260] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.465070 15833 consensus_queue.cc:1059] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:09:09.603458 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.603614 12797 raft_consensus.cc:1240] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20260202 15:09:09.604091 12648 raft_consensus.cc:2393] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 7: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:09.467607 13361 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:09.468371 13326 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:09.473021 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 267113us
W20260202 15:09:09.606019 12690 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.479120 12642 raft_consensus.cc:2393] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 9: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:09.300702 15588 raft_consensus.cc:2749] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:09.530787 13360 raft_consensus.cc:3060] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 7
I20260202 15:09:09.381378 15784 raft_consensus.cc:493] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:09.572749 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.608255 12797 raft_consensus.cc:1240] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:09.583168 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.361545 15834 consensus_queue.cc:1059] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260202 15:09:09.587622 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.617552 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:09.618417 12641 raft_consensus.cc:2410] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260202 15:09:09.520968 15626 raft_consensus.cc:493] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:09.523352 15725 raft_consensus.cc:740] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:09.608100 12791 raft_consensus.cc:1240] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:09.573508 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.608855 13355 raft_consensus.cc:2468] T d91ac50607b34cd1b7d61cd42f7c26d8 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:09.608134 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.621537 12720 raft_consensus.cc:2468] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:09:09.622009 12795 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:09.622330 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.622925 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:09.623561 12795 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:09.609458 12719 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 9.
I20260202 15:09:09.626801 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.627611 12719 raft_consensus.cc:2393] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 4: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:09.609632 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87df3a9146bb41e1ac7468e17d2d9498" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.622942 12720 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:09.630607 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.630651 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:09.632076 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.632057 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.633399 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.634034 12792 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:09.634923 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.636111 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:09.586406 15813 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:09.619431 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.644870 12721 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 5.
I20260202 15:09:09.645041 12765 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:09.642314 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.642987 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:09.646057 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.647434 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.621037 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:09.650025 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.643219 12725 raft_consensus.cc:1240] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:09.642490 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1722) took 1084 ms. Trace:
I20260202 15:09:09.651377 12726 rpcz_store.cc:276] 0202 15:09:08.557730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.557893 (+ 163us) service_pool.cc:225] Handling call
0202 15:09:08.573821 (+ 15928us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:08.700312 (+126491us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:09.642253 (+941941us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400158e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:09.642429 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":263395072}
I20260202 15:09:09.646080 12798 raft_consensus.cc:2368] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5.
I20260202 15:09:09.656476 13359 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 9.
I20260202 15:09:09.656821 13360 raft_consensus.cc:2468] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:09.657910 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11a10c0c64914444a6659e661207c082" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:09.663899 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 202370us
I20260202 15:09:09.664455 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.668121 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.666529 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.669271 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:09.670352 12720 raft_consensus.cc:2393] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 8: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:09.671711 12797 raft_consensus.cc:2393] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 8: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:09:09.644037 15804 consensus_meta.cc:287] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.588s user 0.009s sys 0.000s
I20260202 15:09:09.672155 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.672511 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.690945 12792 raft_consensus.cc:2393] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 7: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
W20260202 15:09:09.673060 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.641640 15626 raft_consensus.cc:515] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.620615 15835 raft_consensus.cc:493] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:09.676738 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.591401 15775 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.695322 12726 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 5.
I20260202 15:09:09.677536 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:09.695715 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.693991 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.677965 12796 raft_consensus.cc:1240] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:09.630228 15815 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:09.682251 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.682895 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.683662 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:09.683745 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:09.678550 13329 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.011s sys 0.027s
I20260202 15:09:09.687145 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:09.684655 13328 leader_election.cc:304] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:09.691550 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.674619 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.605577 15774 raft_consensus.cc:697] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [term 8 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:09.621891 15784 raft_consensus.cc:515] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:09.696148 12721 raft_consensus.cc:2376] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8. Re-sending same reply.
I20260202 15:09:09.678182 13356 raft_consensus.cc:1240] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEADER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:09.694067 12795 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8.
I20260202 15:09:09.664927 15813 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:09.696664 12615 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:10.167234 12798 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:10.168431 12649 raft_consensus.cc:1275] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260202 15:09:09.861524 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:09.702312 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:09.854940 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:10.162973 12690 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.010s sys 0.030s
I20260202 15:09:09.743584 12723 raft_consensus.cc:2393] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:09:10.179610 12690 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.010s sys 0.031s
I20260202 15:09:10.179677 12792 raft_consensus.cc:2468] T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:10.180482 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1755) took 1598 ms. Trace:
I20260202 15:09:10.180800 12645 rpcz_store.cc:276] 0202 15:09:08.581735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.581855 (+ 120us) service_pool.cc:225] Handling call
0202 15:09:08.587042 (+ 5187us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:08.587256 (+ 214us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:08.587283 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:08.587690 (+ 407us) log.cc:844] Serialized 26 byte log entry
0202 15:09:08.752539 (+164849us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:08.752580 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:08.752652 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:09.744187 (+991535us) raft_consensus.cc:1685] finished
0202 15:09:09.744213 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:10.180430 (+436217us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":149789,"threads_started":1,"wal-append.queue_time_us":545775}
W20260202 15:09:10.181774 12690 connection.cc:232] RPC call timeout handler was delayed by -1.0617739409999558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2127s for a response
W20260202 15:09:10.168795 13329 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.009s user 0.031s sys 0.034s
I20260202 15:09:10.170965 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4cc1899a25452c9c57398fe4022645" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:09.685925 15721 raft_consensus.cc:3055] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Stepping down as leader of term 6
W20260202 15:09:09.926090 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:09.980314 12647 consensus_meta.cc:287] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.517s user 0.007s sys 0.000s
W20260202 15:09:10.176734 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 480086us
I20260202 15:09:09.850999 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:09:10.182114 12690 connection.cc:232] RPC call timeout handler was delayed by -1.01144601399983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
I20260202 15:09:10.187469 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62adc5e06e9d428f94bdbd03c58c5e5b" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.188786 12647 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
W20260202 15:09:10.189038 12690 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.516s (SENT) [suppressed 59 similar messages]
I20260202 15:09:10.188243 12615 leader_election.cc:304] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:10.184060 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.190107 12726 raft_consensus.cc:3060] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:10.184248 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.183912 12724 raft_consensus.cc:2410] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260202 15:09:10.191470 12615 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:10.192621 12791 raft_consensus.cc:2368] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5.
W20260202 15:09:10.193617 12690 leader_election.cc:336] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829): Timed out: RequestConsensusVote RPC to 127.11.46.195:45829 timed out after 1.675s (SENT)
I20260202 15:09:10.194357 12690 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:10.185177 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:10.157938 15792 consensus_meta.cc:287] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.959s user 0.007s sys 0.003s
W20260202 15:09:10.199122 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 514494us
I20260202 15:09:10.192087 12794 raft_consensus.cc:2468] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:09.689862 15799 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:09.849010 15732 consensus_meta.cc:287] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.514s user 0.006s sys 0.000s
I20260202 15:09:09.856932 15414 raft_consensus.cc:3055] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 LEADER]: Stepping down as leader of term 8
I20260202 15:09:10.163516 15574 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:09:09.743221 15608 consensus_meta.cc:287] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.136s user 0.008s sys 0.002s
W20260202 15:09:10.185163 15814 consensus_meta.cc:287] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.883s user 0.006s sys 0.001s
I20260202 15:09:10.216717 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.221671 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:10.221768 12799 raft_consensus.cc:2468] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 3.
I20260202 15:09:10.222517 12724 raft_consensus.cc:2393] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 8: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:10.183449 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.198118 12615 leader_election.cc:304] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:10.199221 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.208442 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.214121 12717 raft_consensus.cc:2410] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260202 15:09:10.192198 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:10.223810 12723 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:09.743734 15698 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:09.852140 15726 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.227949 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:10.216804 13326 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.002s sys 0.015s
I20260202 15:09:10.190102 15674 leader_election.cc:290] T 4ed5f5e04d064d6e8d431aa2209cc141 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:10.210999 15663 raft_consensus.cc:2804] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:10.276991 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d68e0d1324241f1878c450b36fac925" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.277539 12792 raft_consensus.cc:2468] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:10.278471 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:09:10.230034 13326 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s user 0.002s sys 0.015s
W20260202 15:09:10.280239 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 79953.2us
W20260202 15:09:10.280362 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 812021us
I20260202 15:09:10.281047 13326 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:10.188103 15800 consensus_meta.cc:287] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.993s user 0.009s sys 0.000s
I20260202 15:09:10.218439 12641 raft_consensus.cc:1275] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260202 15:09:09.874723 15712 leader_election.cc:290] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.301952 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.218696 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.220032 15414 raft_consensus.cc:740] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:09.853108 15662 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.224501 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:10.183543 15845 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260202 15:09:10.196481 15729 consensus_meta.cc:287] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 1.146s user 0.009s sys 0.001s
I20260202 15:09:10.185014 15508 raft_consensus.cc:2749] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260202 15:09:10.067814 15832 log.cc:927] Time spent T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.375s user 0.000s sys 0.000s
I20260202 15:09:10.199294 15830 raft_consensus.cc:2749] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260202 15:09:09.697593 15673 consensus_queue.cc:1059] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.523s
I20260202 15:09:09.692420 15789 consensus_queue.cc:1048] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.525s
I20260202 15:09:10.108436 15756 raft_consensus.cc:2749] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:09.696079 15804 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.183787 15720 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:09.689198 15725 consensus_queue.cc:260] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.187073 15721 raft_consensus.cc:740] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:10.225786 15647 leader_election.cc:290] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.228632 15856 raft_consensus.cc:2749] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:10.358063 15674 raft_consensus.cc:493] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:10.358985 15785 raft_consensus.cc:3055] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:10.358793 15674 raft_consensus.cc:515] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.359541 15785 raft_consensus.cc:740] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:10.360632 15785 consensus_queue.cc:260] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:10.361351 15674 leader_election.cc:290] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.300701 15663 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:10.216723 15792 raft_consensus.cc:515] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.364380 15792 leader_election.cc:290] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.220674 15799 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.331379 15729 raft_consensus.cc:515] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:09.903052 15835 raft_consensus.cc:515] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.226742 15732 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.231230 15814 raft_consensus.cc:515] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.240945 15698 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:10.318559 15861 raft_consensus.cc:493] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:10.333402 15845 raft_consensus.cc:493] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:10.367518 15847 log.cc:927] Time spent T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.180s user 0.000s sys 0.000s
I20260202 15:09:10.356545 15791 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:10.377084 15756 raft_consensus.cc:2749] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:09:10.345026 15862 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:10.368166 15852 log.cc:927] Time spent T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260202 15:09:10.368789 15854 log.cc:927] Time spent T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
W20260202 15:09:10.376955 15825 consensus_meta.cc:287] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.793s user 0.008s sys 0.000s
I20260202 15:09:10.370549 15821 raft_consensus.cc:3055] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:10.356568 15804 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.373672 15813 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.366743 15799 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.309922 15800 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.396526 15845 raft_consensus.cc:515] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.410957 15845 leader_election.cc:290] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.401506 15821 raft_consensus.cc:740] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:10.413770 15821 consensus_queue.cc:260] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.415201 15821 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:10.421809 15835 leader_election.cc:290] T c4798df17a1e4d18a594bb1b37b98bae P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.423362 15814 leader_election.cc:290] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.400486 15862 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.426496 15862 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.395709 15861 raft_consensus.cc:515] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.429231 15861 leader_election.cc:290] T cb4cc1899a25452c9c57398fe4022645 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.402623 15825 raft_consensus.cc:515] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.431787 15825 leader_election.cc:290] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.388012 15849 consensus_queue.cc:1059] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260202 15:09:10.439855 15861 raft_consensus.cc:3055] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:10.440299 15861 raft_consensus.cc:740] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:10.441159 15861 consensus_queue.cc:260] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:10.263386 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:10.477608 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:10.224328 12615 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
W20260202 15:09:10.479830 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 255407us
I20260202 15:09:10.480018 12643 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 in current term 10: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:10.485941 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:58672 (request call id 26) took 2108 ms. Trace:
I20260202 15:09:10.486478 12650 rpcz_store.cc:276] 0202 15:09:08.377469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.378306 (+ 837us) service_pool.cc:225] Handling call
0202 15:09:10.485468 (+2107162us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54000f9bd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed5594d3 00007f24f251713a 00007f24ebab2934 00007f24ebab28c2 00007f24ebab27f1 00007f24ebab276c 00007f24ebab1a22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:10.485681 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":919872896}
I20260202 15:09:10.492268 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34792 (request call id 1741) took 1467 ms. Trace:
I20260202 15:09:10.492690 12648 rpcz_store.cc:276] 0202 15:09:09.024701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.024830 (+ 129us) service_pool.cc:225] Handling call
0202 15:09:10.491701 (+1466871us) spinlock_profiling.cc:243] Waited 546 ms on lock 0x7b54000799d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:10.492008 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1256809216}
I20260202 15:09:10.499169 12544 catalog_manager.cc:5654] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:09:10.503014 12544 catalog_manager.cc:5654] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:09:10.504561 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1741) took 5292 ms. Trace:
I20260202 15:09:10.504866 12689 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:10.505225 12544 catalog_manager.cc:5654] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 5, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 5 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:09:10.506177 12689 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:10.507256 12544 catalog_manager.cc:5654] T af412f1b23854fdea68917d1df084a35 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 3 to 5, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 5 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
W20260202 15:09:10.508440 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 314729us
I20260202 15:09:10.508606 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1770) took 1153 ms. Trace:
I20260202 15:09:10.509338 12641 rpcz_store.cc:276] 0202 15:09:09.354856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.643291 (+288435us) service_pool.cc:225] Handling call
0202 15:09:09.643521 (+ 230us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:10.301776 (+658255us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.508025 (+206249us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b540024e854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:10.508412 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1314239744}
I20260202 15:09:10.509534 12544 catalog_manager.cc:5654] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 3 to 6, leader changed from 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 6 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:09:10.511942 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 520) took 1212 ms. Trace:
I20260202 15:09:10.505035 12644 rpcz_store.cc:276] 0202 15:09:05.211873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:07.261629 (+2049756us) service_pool.cc:225] Handling call
0202 15:09:07.261880 (+ 251us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:07.296949 (+ 35069us) raft_consensus.cc:1534] Early marking committed up to index 8
0202 15:09:08.171104 (+874155us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.614868 (+443764us) log.cc:844] Serialized 14 byte log entry
0202 15:09:08.719733 (+104865us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:08.719798 (+ 65us) raft_consensus.cc:1645] Marking committed up to 8
0202 15:09:08.719855 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.503906 (+1784051us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b54001de7d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:10.504003 (+ 97us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:10.504331 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13135,"prepare.queue_time_us":840764,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":7374,"raft.queue_time_us":232425,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":19211,"spinlock_wait_cycles":80198016,"thread_start_us":1349932,"threads_started":3,"wal-append.queue_time_us":529995,"wal-append.run_cpu_time_us":2660,"wal-append.run_wall_time_us":1323905}
I20260202 15:09:10.512739 12643 rpcz_store.cc:276] 0202 15:09:09.299595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.299997 (+ 402us) service_pool.cc:225] Handling call
0202 15:09:10.511714 (+1211717us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540008a2d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:10.511888 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2416135936}
I20260202 15:09:10.514041 12648 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "6921687ed3814cccb2bd9c606875b9f0"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:09:10.534235 12643 raft_consensus.cc:1240] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:10.536252 12726 raft_consensus.cc:2468] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:10.536988 12650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:10.538183 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:10.521757 12719 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 7: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:10.522236 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1869) took 1131 ms. Trace:
I20260202 15:09:10.522893 12720 raft_consensus.cc:2393] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 10: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:10.523658 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1718) took 3080 ms. Trace:
I20260202 15:09:10.523840 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1720) took 2431 ms. Trace:
I20260202 15:09:10.525614 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed220e8ce3ce45ff8b531267a15de19e" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:10.538965 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:10.540722 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:10.541672 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:10.547367 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:10.551558 12649 raft_consensus.cc:1240] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: []
W20260202 15:09:10.555920 12763 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:10.219328 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.557200 12795 raft_consensus.cc:2376] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8. Re-sending same reply.
W20260202 15:09:10.546602 12545 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45944 (request call id 310) took 20013 ms (client timeout 15000 ms). Trace:
I20260202 15:09:10.560304 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.308876 12798 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
W20260202 15:09:10.560796 12793 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56734 (request call id 1643) took 3026 ms (client timeout 1755 ms). Trace:
I20260202 15:09:10.560643 12544 catalog_manager.cc:5654] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 6 to 8, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195), now has a pending config: VOTER bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193), VOTER 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194), VOTER 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195), VOTER 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 8 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: NON_VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
W20260202 15:09:10.565480 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:10.566118 12796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:10.742846 12798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1950) took 1084 ms. Trace:
I20260202 15:09:10.743273 12798 rpcz_store.cc:276] 0202 15:09:09.658865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.663340 (+ 4475us) service_pool.cc:225] Handling call
0202 15:09:10.742637 (+1079297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:10.744387 12765 leader_election.cc:304] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:10.744505 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3e391e37e424b1b8b2c63352ec8c198" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:10.745574 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:10.571403 12799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:10.299849 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:10.520185 12690 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:10.317826 13328 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:10.229588 15608 raft_consensus.cc:515] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:10.366667 15676 leader_election.cc:290] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.574106 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44940 (request call id 911) took 1298 ms. Trace:
I20260202 15:09:10.577240 15698 leader_election.cc:290] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.577422 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:10.578725 13363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 915) took 1301 ms. Trace:
I20260202 15:09:10.579759 15417 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:10.580525 13359 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260202 15:09:10.583084 15407 raft_consensus.cc:2749] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260202 15:09:10.584201 13357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 907) took 3112 ms. Trace:
I20260202 15:09:10.584100 15414 consensus_queue.cc:260] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.586169 13356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 909) took 1310 ms. Trace:
I20260202 15:09:10.587127 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e78bd8179024ae0aca19878b91342a9" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:10.588991 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:10.352387 15626 leader_election.cc:290] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.319077 15662 raft_consensus.cc:2749] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:10.362529 15663 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:09:10.593580 15706 consensus_meta.cc:287] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 1.248s user 0.014s sys 0.001s
I20260202 15:09:10.342028 15712 raft_consensus.cc:493] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:10.354569 15721 consensus_queue.cc:260] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:10.371495 15729 leader_election.cc:290] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.596903 15732 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.356804 15725 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 4 FOLLOWER]: Advancing to term 5
I20260202 15:09:10.604727 15775 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.605141 15540 raft_consensus.cc:3055] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:10.315716 15774 consensus_queue.cc:237] T 2ac5476ea30a46548db9ede417d946e6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.607229 12718 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 7.
I20260202 15:09:10.609268 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:10.616873 15720 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader election won for term 9
I20260202 15:09:10.616573 15503 consensus_queue.cc:1048] T c4798df17a1e4d18a594bb1b37b98bae P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260202 15:09:10.628239 15572 consensus_queue.cc:1048] T d91ac50607b34cd1b7d61cd42f7c26d8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260202 15:09:10.632681 15784 leader_election.cc:290] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:10.633069 15569 consensus_queue.cc:1059] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260202 15:09:10.634511 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:10.634975 15785 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
W20260202 15:09:10.647357 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 158623us
I20260202 15:09:10.649537 15800 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.663714 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:10.705643 15801 raft_consensus.cc:2804] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:10.711346 12725 rpcz_store.cc:276] 0202 15:09:08.092157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.403499 (+311342us) service_pool.cc:225] Handling call
0202 15:09:08.442376 (+ 38877us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:09.651090 (+1208714us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.523294 (+872204us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540033bb54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:10.523647 (+ 353us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2761193856}
I20260202 15:09:10.710491 12717 rpcz_store.cc:276] 0202 15:09:09.390345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.572488 (+182143us) service_pool.cc:225] Handling call
0202 15:09:10.521700 (+949212us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b5400145854. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:10.521945 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1391606528}
I20260202 15:09:10.728035 12641 raft_consensus.cc:3060] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:10.731361 12643 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 10.
I20260202 15:09:10.732432 12721 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:10.732702 15807 consensus_queue.cc:1059] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260202 15:09:10.733124 12645 raft_consensus.cc:1240] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 6. Current term is 7. Ops: [5.4-6.5]
W20260202 15:09:10.733453 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:10.734229 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 228222us
I20260202 15:09:10.735445 15806 raft_consensus.cc:2804] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:10.733953 12647 raft_consensus.cc:3060] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Advancing to term 10
I20260202 15:09:10.735983 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:10.722667 12545 rpcz_store.cc:269] 0202 15:08:50.533052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:08:50.533284 (+ 232us) service_pool.cc:225] Handling call
0202 15:09:10.546369 (+20013085us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0202 15:09:01.093693 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0202 15:09:01.094532 (+ 839us) write_op.cc:432] Acquiring schema lock in shared mode
0202 15:09:01.094591 (+ 59us) write_op.cc:435] Acquired schema lock
0202 15:09:01.094613 (+ 22us) tablet.cc:660] Decoding operations
0202 15:09:01.220534 (+125921us) write_op.cc:620] Acquiring the partition lock for write op
0202 15:09:01.220610 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0202 15:09:01.220631 (+ 21us) tablet.cc:683] Acquiring locks for 9 operations
0202 15:09:01.221283 (+ 652us) tablet.cc:699] Row locks acquired
0202 15:09:01.221321 (+ 38us) write_op.cc:260] PREPARE: finished
0202 15:09:01.221471 (+ 150us) write_op.cc:270] Start()
0202 15:09:01.221612 (+ 141us) write_op.cc:276] Timestamp: P: 1770044941221418 usec, L: 0
0202 15:09:01.221642 (+ 30us) op_driver.cc:348] REPLICATION: starting
0202 15:09:01.222220 (+ 578us) log.cc:844] Serialized 3626 byte log entry
0202 15:09:02.359417 (+1137197us) op_driver.cc:464] REPLICATION: finished
0202 15:09:02.548261 (+188844us) write_op.cc:301] APPLY: starting
0202 15:09:02.554485 (+ 6224us) tablet.cc:1365] starting BulkCheckPresence
0202 15:09:02.566086 (+ 11601us) tablet.cc:1368] finished BulkCheckPresence
0202 15:09:02.566216 (+ 130us) tablet.cc:1370] starting ApplyRowOperation cycle
0202 15:09:02.573447 (+ 7231us) tablet.cc:1381] finished ApplyRowOperation cycle
0202 15:09:02.581486 (+ 8039us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0202 15:09:02.585916 (+ 4430us) write_op.cc:312] APPLY: finished
0202 15:09:02.599086 (+ 13170us) log.cc:844] Serialized 71 byte log entry
0202 15:09:02.599242 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:09:02.599349 (+ 107us) write_op.cc:454] Released schema lock
0202 15:09:02.599401 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0202 15:09:02.599735 (+ 334us) write_op.cc:489] Releasing partition, row and schema locks
0202 15:09:02.599759 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1488610,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":160632,"apply.run_cpu_time_us":5830,"apply.run_wall_time_us":51491,"num_ops":9,"prepare.queue_time_us":534923,"prepare.run_cpu_time_us":81796,"prepare.run_wall_time_us":363153,"raft.queue_time_us":469746,"raft.run_cpu_time_us":17286,"raft.run_wall_time_us":102342,"replication_time_us":1137623,"thread_start_us":1018958,"threads_started":4,"wal-append.queue_time_us":331782,"wal-append.run_cpu_time_us":38319,"wal-append.run_wall_time_us":2041975}]]}
I20260202 15:09:10.874225 13355 rpcz_store.cc:276] 0202 15:09:09.275971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.276098 (+ 127us) service_pool.cc:225] Handling call
0202 15:09:10.573644 (+1297546us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54006090d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:10.573857 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":520815360}
I20260202 15:09:10.738224 12644 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260202 15:09:10.738529 13354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 922) took 1166 ms. Trace:
I20260202 15:09:10.740325 12797 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1949) took 1098 ms. Trace:
I20260202 15:09:10.740919 12792 raft_consensus.cc:1275] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Refusing update from remote peer bbf140958c3b4f339c512a9dc4be2dab: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260202 15:09:10.737823 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:10.742731 12646 raft_consensus.cc:2410] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260202 15:09:10.731614 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:10.872362 12763 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:10.746572 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:10.749178 12723 raft_consensus.cc:1240] T d30f8f27ce00483c85fb1758a4f81f03 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:10.747206 12794 raft_consensus.cc:2410] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260202 15:09:10.711266 12722 rpcz_store.cc:276] 0202 15:09:07.443071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.393450 (+950379us) service_pool.cc:225] Handling call
0202 15:09:08.393735 (+ 285us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:08.394062 (+ 327us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:08.394093 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:08.407744 (+ 13651us) log.cc:844] Serialized 26 byte log entry
0202 15:09:10.000189 (+1592445us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b1800389140. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471 00007f24ed51a73f
0202 15:09:10.278623 (+278434us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:10.278853 (+ 230us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.278919 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:10.522876 (+243957us) raft_consensus.cc:1685] finished
0202 15:09:10.522912 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:10.523416 (+ 504us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5242,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":6356,"spinlock_wait_cycles":10566016,"thread_start_us":276929,"threads_started":1,"wal-append.queue_time_us":1965672}
I20260202 15:09:10.748849 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1733) took 1131 ms. Trace:
I20260202 15:09:10.871474 13357 rpcz_store.cc:276] 0202 15:09:07.471625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.369193 (+897568us) service_pool.cc:225] Handling call
0202 15:09:08.369532 (+ 339us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:08.869374 (+499842us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.583538 (+1714164us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54006af754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:10.583983 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1134484352}
I20260202 15:09:10.875243 13360 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:10.747081 12800 raft_consensus.cc:2376] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7. Re-sending same reply.
W20260202 15:09:10.737286 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 418991us
W20260202 15:09:10.886276 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:10.889724 15831 raft_consensus.cc:493] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:10.890751 15873 raft_consensus.cc:2804] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:09:10.891747 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1857) took 1504 ms. Trace:
W20260202 15:09:10.891961 15887 log.cc:927] Time spent T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
I20260202 15:09:10.884737 15801 raft_consensus.cc:493] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:10.897605 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:10.897856 15817 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:10.899231 12642 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 8: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:10.899291 15837 raft_consensus.cc:3055] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:10.899744 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3a3797f335d43dca87344d8954a1158" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:09:10.902187 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:10.911325 12724 raft_consensus.cc:1240] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 9. Current term is 10. Ops: []
I20260202 15:09:10.750037 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:10.871701 13356 rpcz_store.cc:276] 0202 15:09:09.275469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.277996 (+ 2527us) service_pool.cc:225] Handling call
0202 15:09:09.278251 (+ 255us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:10.168574 (+890323us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.585647 (+417073us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b540016c1d4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:10.586014 (+ 367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":919279360}
W20260202 15:09:10.742604 12793 rpcz_store.cc:269] 0202 15:09:07.532447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:08.116749 (+584302us) service_pool.cc:225] Handling call
0202 15:09:10.558859 (+2442110us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540004c754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:10.559112 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2770901760}
I20260202 15:09:10.876979 15540 raft_consensus.cc:740] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:10.905450 12719 rpcz_store.cc:276] 0202 15:09:09.617645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.669061 (+ 51416us) service_pool.cc:225] Handling call
0202 15:09:10.748541 (+1079480us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b5400145354. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:10.748744 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1957598464}
I20260202 15:09:10.911950 12720 rpcz_store.cc:276] 0202 15:09:09.387479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.686965 (+299486us) service_pool.cc:225] Handling call
0202 15:09:10.891485 (+1204520us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54002ff654. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:10.891684 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":810604928}
I20260202 15:09:10.743613 12544 catalog_manager.cc:5654] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 5 to 8, leader changed from 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196) to bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193). New cstate: current_term: 8 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260202 15:09:10.933391 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:10.934983 15817 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:10.937038 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 417047us
I20260202 15:09:10.927455 15837 raft_consensus.cc:740] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:10.938833 12690 leader_election.cc:304] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:10.939185 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:10.937095 15817 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:10.941355 12641 raft_consensus.cc:2468] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 9.
I20260202 15:09:10.888733 12797 rpcz_store.cc:276] 0202 15:09:09.641472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.686177 (+ 44705us) service_pool.cc:225] Handling call
0202 15:09:09.760281 (+ 74104us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:09.760463 (+ 182us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:09:10.558857 (+798394us) log.cc:844] Serialized 14 byte log entry
0202 15:09:10.739528 (+180671us) log.cc:844] Serialized 14 byte log entry
0202 15:09:10.739730 (+ 202us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:10.739817 (+ 87us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:09:10.739923 (+ 106us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.740018 (+ 95us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:10.740148 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2396,"prepare.queue_time_us":662534,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":89,"raft.queue_time_us":3890,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":10973,"thread_start_us":531947,"threads_started":2}
I20260202 15:09:10.949805 12800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56722 (request call id 1969) took 1091 ms. Trace:
I20260202 15:09:10.948378 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:10.953768 12722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:10.873905 13363 rpcz_store.cc:276] 0202 15:09:09.276981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.279316 (+ 2335us) service_pool.cc:225] Handling call
0202 15:09:09.296556 (+ 17240us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:09.296926 (+ 370us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:09.296961 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:09.316900 (+ 19939us) log.cc:844] Serialized 26 byte log entry
0202 15:09:09.696660 (+379760us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:09:09.696695 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:09.696781 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:10.578198 (+881417us) raft_consensus.cc:1685] finished
0202 15:09:10.578231 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:10.578490 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":536,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":19647,"thread_start_us":362431,"threads_started":1,"wal-append.queue_time_us":1012763}
I20260202 15:09:10.954634 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d8302d577042feaa44f63d596dc87d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:10.924872 15801 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:10.959654 12722 raft_consensus.cc:2410] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260202 15:09:10.960249 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:10.960405 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:10.958909 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:10.956924 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:10.886886 13354 rpcz_store.cc:276] 0202 15:09:09.571676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.572226 (+ 550us) service_pool.cc:225] Handling call
0202 15:09:09.588517 (+ 16291us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:10.067221 (+478704us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:10.067252 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:10.067675 (+ 423us) log.cc:844] Serialized 26 byte log entry
0202 15:09:10.578096 (+510421us) spinlock_profiling.cc:243] Waited 60.3 ms on lock 0x7b18003bb360. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471 00007f24ed51a73f
0202 15:09:10.660590 (+ 82494us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:10.720865 (+ 60275us) log.cc:844] Serialized 14 byte log entry
0202 15:09:10.726375 (+ 5510us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.729079 (+ 2704us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:10.729137 (+ 58us) raft_consensus.cc:1685] finished
0202 15:09:10.738146 (+ 9009us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:10.738372 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":90443,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":37,"raft.queue_time_us":415054,"raft.run_cpu_time_us":476,"raft.run_wall_time_us":97461,"spinlock_wait_cycles":138611584,"thread_start_us":618793,"threads_started":3,"wal-append.queue_time_us":541737}
I20260202 15:09:10.962457 12795 raft_consensus.cc:1240] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:10.965512 12647 raft_consensus.cc:2468] T 323d926990494ad1aafb6aa0282d9f68 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 10.
I20260202 15:09:10.967816 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:10.955829 12800 rpcz_store.cc:276] 0202 15:09:09.858010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.227215 (+369205us) service_pool.cc:225] Handling call
0202 15:09:10.934291 (+707076us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:10.961665 15720 raft_consensus.cc:697] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:10.969636 12791 raft_consensus.cc:1240] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260202 15:09:10.969930 15800 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:10.970091 13357 raft_consensus.cc:2368] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8.
I20260202 15:09:10.973899 12717 raft_consensus.cc:1240] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:10.973760 12615 leader_election.cc:304] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:10.974721 15791 raft_consensus.cc:493] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
I20260202 15:09:10.974813 15873 raft_consensus.cc:697] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:10.975417 15813 consensus_queue.cc:1059] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260202 15:09:10.976245 15811 consensus_queue.cc:1059] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260202 15:09:10.977448 15804 consensus_queue.cc:1059] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20260202 15:09:10.979247 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 44210.2us
I20260202 15:09:11.020149 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:10.979542 15736 raft_consensus.cc:3055] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Stepping down as leader of term 8
I20260202 15:09:10.979656 15572 consensus_queue.cc:1059] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260202 15:09:10.980612 12798 raft_consensus.cc:2393] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 8: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:10.980542 13363 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:10.981212 15732 raft_consensus.cc:2804] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260202 15:09:10.981971 13361 raft_consensus.cc:2468] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5.
I20260202 15:09:10.911435 15831 raft_consensus.cc:515] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:10.982965 12544 catalog_manager.cc:5654] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 4 to 7, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 7 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:09:10.991222 12721 raft_consensus.cc:2468] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:11.037551 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37612 (request call id 1731) took 1432 ms. Trace:
I20260202 15:09:10.939905 15806 raft_consensus.cc:697] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:11.001190 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 459) took 1602 ms. Trace:
I20260202 15:09:11.009177 12799 raft_consensus.cc:2468] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:11.014124 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 456) took 1646 ms. Trace:
I20260202 15:09:11.002417 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:11.003489 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:11.020913 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1856) took 1623 ms. Trace:
I20260202 15:09:11.013604 13355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:10.916960 15706 raft_consensus.cc:515] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.031850 12641 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a776a09df954aa6ac5c756e8da6c56a" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:10.974457 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 656542us
W20260202 15:09:11.033697 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.033412 15732 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:11.050945 15732 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Advancing to term 10
I20260202 15:09:11.051702 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:11.051968 15608 leader_election.cc:290] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:11.052124 12798 raft_consensus.cc:1240] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260202 15:09:11.052312 12643 raft_consensus.cc:2468] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 7.
I20260202 15:09:11.052845 12645 raft_consensus.cc:1240] T a3e391e37e424b1b8b2c63352ec8c198 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:11.053166 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:11.038283 12724 rpcz_store.cc:276] 0202 15:09:09.385922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.232680 (+846758us) service_pool.cc:225] Handling call
0202 15:09:10.893340 (+660660us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:10.939720 (+ 46380us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.988418 (+ 48698us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b54002ff650. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:10.988559 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1351445760}
W20260202 15:09:11.043180 12614 leader_election.cc:336] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Timed out: RequestConsensusVote RPC to 127.11.46.196:36191 timed out after 1.504s (SENT)
I20260202 15:09:11.055368 12614 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:11.055550 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 116891us
I20260202 15:09:11.056514 12690 leader_election.cc:304] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:11.056948 12614 leader_election.cc:336] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Timed out: RequestConsensusVote RPC to 127.11.46.196:36191 timed out after 1.512s (SENT)
I20260202 15:09:11.058065 12794 raft_consensus.cc:1240] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260202 15:09:11.058854 12722 raft_consensus.cc:1275] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260202 15:09:11.059015 15845 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:11.040987 12649 raft_consensus.cc:1240] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:11.060590 13361 raft_consensus.cc:1275] T e4d8302d577042feaa44f63d596dc87d P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260202 15:09:11.060830 15572 raft_consensus.cc:3055] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:11.061242 15572 raft_consensus.cc:740] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:11.061643 12792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56722 (request call id 1964) took 1207 ms. Trace:
I20260202 15:09:11.037863 12721 rpcz_store.cc:276] 0202 15:09:09.604700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.170149 (+565449us) service_pool.cc:225] Handling call
0202 15:09:10.525192 (+355043us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x7b58001d7098. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ea2f18b1 00007f24ea473db1 00007f24ea473c62 00007f24f331edd9 00007f24f249e260 00007f24f2566dbb 00007f24f2566e41 00007f24f2514bd0 00007f24f25144b4 00007f24ebaa4834 00007f24ebaa47c2
0202 15:09:11.037494 (+512302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18660736}
W20260202 15:09:11.043139 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 69526.9us
I20260202 15:09:11.063714 12721 pending_rounds.cc:85] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Aborting all ops after (but not including) 3
I20260202 15:09:11.043548 15811 raft_consensus.cc:3060] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:11.063999 12721 pending_rounds.cc:107] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260202 15:09:11.064486 12721 raft_consensus.cc:2889] T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:11.064754 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:09:11.065966 12718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59480 (request call id 454) took 1712 ms (client timeout 1513 ms). Trace:
W20260202 15:09:11.045416 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.043390 15662 raft_consensus.cc:3055] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:11.047613 12689 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:11.066865 15662 raft_consensus.cc:740] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:11.048877 13363 raft_consensus.cc:1240] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260202 15:09:11.047192 12720 raft_consensus.cc:1217] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 3.4->[4.5-7.7] Dedup: 4.5->[4.6-7.7]
I20260202 15:09:11.068004 15662 consensus_queue.cc:260] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:11.054461 15806 consensus_queue.cc:237] T d3a3797f335d43dca87344d8954a1158 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.069417 15662 raft_consensus.cc:3060] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:11.021353 15417 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.059260 12722 pending_rounds.cc:85] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Aborting all ops after (but not including) 8
I20260202 15:09:11.061987 12792 rpcz_store.cc:276] 0202 15:09:09.853744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.278129 (+424385us) service_pool.cc:225] Handling call
0202 15:09:10.740202 (+462073us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:10.888825 (+148623us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.061441 (+172616us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b5400116550. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:11.061565 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":973275008}
I20260202 15:09:11.076496 12722 pending_rounds.cc:107] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260202 15:09:11.076809 12722 raft_consensus.cc:2889] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:11.039196 12717 rpcz_store.cc:276] 0202 15:09:09.387240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.973323 (+1586083us) service_pool.cc:225] Handling call
0202 15:09:10.973658 (+ 335us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.010468 (+ 36810us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.010677 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:11.064867 15861 raft_consensus.cc:3060] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:09:11.066352 12718 rpcz_store.cc:269] 0202 15:09:09.353479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.382456 (+ 28977us) service_pool.cc:225] Handling call
0202 15:09:11.065756 (+1683300us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400146254. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:11.065889 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2474036608}
I20260202 15:09:11.028322 15736 raft_consensus.cc:740] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:11.038651 12723 rpcz_store.cc:276] 0202 15:09:09.353894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.226289 (+872395us) service_pool.cc:225] Handling call
0202 15:09:10.748785 (+522496us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:10.929579 (+180794us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:10.995409 (+ 65830us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b5400145350. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:11.000406 (+ 4997us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1200856832}
I20260202 15:09:10.974413 15821 consensus_queue.cc:1048] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260202 15:09:11.055859 12724 raft_consensus.cc:1240] T 1c6861781ee34db9b79870494af1cced P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:11.064487 12615 leader_election.cc:304] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:11.038069 13362 raft_consensus.cc:2393] T 0f70fb0b2ffc4a51b06fe26a22355100 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 8: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:11.068758 12720 pending_rounds.cc:85] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Aborting all ops after (but not including) 5
W20260202 15:09:11.098162 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 57583.9us
I20260202 15:09:11.098359 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1725) took 1501 ms. Trace:
I20260202 15:09:11.098501 12720 pending_rounds.cc:107] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260202 15:09:11.099071 12720 raft_consensus.cc:2889] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:11.098757 12724 rpcz_store.cc:276] 0202 15:09:09.597191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.055206 (+1458015us) service_pool.cc:225] Handling call
0202 15:09:11.055551 (+ 345us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.098164 (+ 42613us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.098315 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:09:11.106648 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.121459 12795 raft_consensus.cc:1240] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260202 15:09:11.036114 15569 raft_consensus.cc:2749] T c4798df17a1e4d18a594bb1b37b98bae P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:11.135535 12643 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260202 15:09:11.134003 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 57372.1us
I20260202 15:09:10.939240 15837 consensus_queue.cc:260] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.062121 15572 consensus_queue.cc:260] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.116230 12544 catalog_manager.cc:5654] T a739ed9c340d4dd9935861456bc3f3d3 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 5, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 5 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:09:11.158370 12641 raft_consensus.cc:2393] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 7: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:11.015043 15800 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.129887 15813 raft_consensus.cc:3060] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:11.162909 13359 raft_consensus.cc:2376] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 5. Re-sending same reply.
W20260202 15:09:11.170502 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42782 (request call id 513) took 2028 ms (client timeout 2152 ms). Trace:
I20260202 15:09:11.168010 13355 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
W20260202 15:09:11.183862 12646 rpcz_store.cc:269] 0202 15:09:09.142409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:09.216005 (+ 73596us) service_pool.cc:225] Handling call
0202 15:09:11.149258 (+1933253us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b540020e754. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:11.170442 (+ 21184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3374350592}
I20260202 15:09:11.187516 12647 raft_consensus.cc:1240] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:11.168010 15407 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260202 15:09:11.196116 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1734) took 1573 ms. Trace:
I20260202 15:09:11.196173 12765 leader_election.cc:304] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:11.196517 12723 rpcz_store.cc:276] 0202 15:09:09.623103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.087223 (+1464120us) service_pool.cc:225] Handling call
0202 15:09:11.195515 (+108292us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.195794 (+ 279us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:11.195832 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:11.195878 (+ 46us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:11.195906 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.195951 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:11.196043 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:11.197777 12723 raft_consensus.cc:1240] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:11.198243 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1736) took 1499 ms. Trace:
W20260202 15:09:11.195163 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.135560 15833 raft_consensus.cc:2749] T af412f1b23854fdea68917d1df084a35 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:11.075389 15791 raft_consensus.cc:515] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.198515 12723 rpcz_store.cc:276] 0202 15:09:09.699068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.197314 (+1498246us) service_pool.cc:225] Handling call
0202 15:09:11.197586 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.198050 (+ 464us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.198177 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:09:11.195227 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 88210.6us
I20260202 15:09:11.188174 13328 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:11.212505 12800 raft_consensus.cc:1240] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260202 15:09:11.216368 12718 raft_consensus.cc:1240] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
W20260202 15:09:11.216635 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 169106us
I20260202 15:09:11.212432 12763 leader_election.cc:304] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:11.218288 12644 raft_consensus.cc:3055] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:11.218755 12644 raft_consensus.cc:740] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:11.212847 15721 raft_consensus.cc:3060] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:11.179497 15845 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:11.225540 12649 raft_consensus.cc:1275] T 87df3a9146bb41e1ac7468e17d2d9498 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260202 15:09:11.171411 15831 leader_election.cc:290] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:11.229240 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59480 (request call id 462) took 1747 ms. Trace:
I20260202 15:09:11.233198 12722 rpcz_store.cc:276] 0202 15:09:09.469785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.995667 (+1525882us) service_pool.cc:225] Handling call
0202 15:09:11.033561 (+ 37894us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.216542 (+182981us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.216700 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:11.234498 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1732) took 1616 ms. Trace:
I20260202 15:09:11.234932 12718 rpcz_store.cc:276] 0202 15:09:09.612392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.097576 (+1485184us) service_pool.cc:225] Handling call
0202 15:09:11.182189 (+ 84613us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.216711 (+ 34522us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.228987 (+ 12276us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260202 15:09:11.260572 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 189848us
I20260202 15:09:11.245419 15789 raft_consensus.cc:2804] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:11.243711 15837 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:11.243150 15706 leader_election.cc:290] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:11.237300 15736 consensus_queue.cc:260] T f72d171c95a149d890432428ecf80739 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.279358 15508 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.138s
I20260202 15:09:11.191588 15873 consensus_queue.cc:237] T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:11.281054 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 216749us
I20260202 15:09:11.294966 12764 leader_election.cc:400] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8.
I20260202 15:09:11.290720 15808 raft_consensus.cc:493] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:11.228672 15712 raft_consensus.cc:515] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:11.307283 12726 raft_consensus.cc:2468] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 7.
I20260202 15:09:11.251924 15870 raft_consensus.cc:2764] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260202 15:09:11.295280 12764 leader_election.cc:403] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260202 15:09:11.308251 15712 leader_election.cc:290] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:09:11.308717 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 272289us
I20260202 15:09:11.291306 15503 raft_consensus.cc:2749] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260202 15:09:11.311528 15503 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:11.298302 15417 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:11.315183 12644 consensus_queue.cc:260] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:11.298885 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.299717 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
W20260202 15:09:11.300184 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 130445us
W20260202 15:09:11.318961 12688 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:11.301998 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 89642.9us
I20260202 15:09:11.303383 15791 leader_election.cc:290] T 4d879eebd1924ad8b6f65647f4e73362 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:11.304463 15857 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:11.321177 15857 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:11.321534 15857 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:11.305547 12615 leader_election.cc:304] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:11.322625 15791 raft_consensus.cc:2749] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:11.323129 12615 leader_election.cc:304] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:11.323906 15791 raft_consensus.cc:2804] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:11.324342 15791 raft_consensus.cc:493] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:11.324080 15663 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.324765 15791 raft_consensus.cc:3060] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:11.325032 12648 raft_consensus.cc:606] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Received request to transfer leadership to TS 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:11.325332 12615 leader_election.cc:400] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5.
I20260202 15:09:11.325409 12648 raft_consensus.cc:614] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260202 15:09:11.325604 12615 leader_election.cc:403] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260202 15:09:11.315207 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.326395 15815 raft_consensus.cc:2749] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829) with higher term. Message: Invalid argument: T 62adc5e06e9d428f94bdbd03c58c5e5b P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab for earlier term 4. Current term is 5.
I20260202 15:09:11.327119 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:11.327817 12615 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:11.328743 15815 raft_consensus.cc:2804] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:11.329144 15815 raft_consensus.cc:493] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:11.329505 15815 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:11.335387 12615 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:11.336539 15663 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:11.340479 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.342288 12724 raft_consensus.cc:1240] T ed220e8ce3ce45ff8b531267a15de19e P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260202 15:09:11.316818 12544 catalog_manager.cc:5654] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 3, leader changed from 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 3 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:09:11.338114 15815 raft_consensus.cc:515] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:11.357040 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 65777.5us
I20260202 15:09:11.237264 15845 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.212409 15720 consensus_queue.cc:237] T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.364207 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:11.314531 15789 raft_consensus.cc:697] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:11.363874 13328 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:11.377724 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.357527 12690 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:11.391252 12722 raft_consensus.cc:1240] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 8. Current term is 9. Ops: []
I20260202 15:09:11.393096 13360 raft_consensus.cc:2468] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:09:11.401257 13361 raft_consensus.cc:1240] T 488d33b86df249748c59fbf945f4e8b3 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:11.272140 15849 consensus_queue.cc:1059] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.260s
I20260202 15:09:11.393778 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1855) took 1993 ms. Trace:
I20260202 15:09:11.408763 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:11.417536 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 44514.8us
I20260202 15:09:11.419395 12764 leader_election.cc:304] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:11.408404 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9592591cd602455aa52fdc1d84e48b63" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:11.363741 15508 consensus_queue.cc:1243] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer bbf140958c3b4f339c512a9dc4be2dab log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260202 15:09:11.434010 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 196077us
W20260202 15:09:11.426129 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 79304.9us
I20260202 15:09:11.467162 12723 raft_consensus.cc:1275] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260202 15:09:11.467566 12723 pending_rounds.cc:85] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: Aborting all ops after (but not including) 4
I20260202 15:09:11.473703 12723 pending_rounds.cc:107] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260202 15:09:11.382282 15663 raft_consensus.cc:2804] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260202 15:09:11.473976 12723 raft_consensus.cc:2889] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:11.473160 12544 catalog_manager.cc:5654] T 21bc61b2f0f64d0b80e5d98615b1d6b6 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 3 to 5, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 5 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:09:11.425122 15417 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:11.416554 15821 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:11.486948 12722 raft_consensus.cc:1275] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260202 15:09:11.491413 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.404138 12719 rpcz_store.cc:276] 0202 15:09:09.386995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.935420 (+1548425us) service_pool.cc:225] Handling call
0202 15:09:10.935825 (+ 405us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:10.936134 (+ 309us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:09:11.027620 (+ 91486us) log.cc:844] Serialized 14 byte log entry
0202 15:09:11.364948 (+337328us) log.cc:844] Serialized 14 byte log entry
0202 15:09:11.379888 (+ 14940us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:11.379948 (+ 60us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:09:11.380004 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.380076 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:11.380203 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":178044,"prepare.queue_time_us":313338,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":7191,"raft.queue_time_us":47290,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":56379,"thread_start_us":181775,"threads_started":2}
I20260202 15:09:11.372076 15801 raft_consensus.cc:515] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:11.508934 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 103244us
W20260202 15:09:11.508116 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 220991us
I20260202 15:09:11.478729 15849 consensus_queue.cc:1243] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer 6921687ed3814cccb2bd9c606875b9f0 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260202 15:09:11.529348 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 181709us
I20260202 15:09:11.478953 15663 raft_consensus.cc:493] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting leader election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:11.401304 15712 consensus_queue.cc:1059] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.107s
I20260202 15:09:11.535454 12650 raft_consensus.cc:1240] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:11.498241 15743 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.122s
I20260202 15:09:11.530922 15800 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:11.540746 12688 leader_election.cc:304] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:11.553025 12544 catalog_manager.cc:5654] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 4 to 8, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 8 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:09:11.479496 15873 consensus_queue.cc:1059] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260202 15:09:11.548552 13326 leader_election.cc:304] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:11.563428 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 223706us
I20260202 15:09:11.534150 15663 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:09:11.565972 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 67217.5us
I20260202 15:09:11.566648 12764 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:11.569685 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1740) took 1002 ms. Trace:
I20260202 15:09:11.575848 12689 leader_election.cc:304] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:09:11.574891 15789 consensus_queue.cc:237] T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:11.580124 15712 consensus_queue.cc:1243] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer 66b8c715b9e6479e9218b08bd3156380 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:09:11.580669 15712 raft_consensus.cc:3055] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:11.580952 15712 raft_consensus.cc:740] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:09:11.582011 15662 consensus_meta.cc:287] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.505s user 0.005s sys 0.002s
I20260202 15:09:11.582077 15774 raft_consensus.cc:2749] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:11.583585 12795 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1672) took 1015 ms. Trace:
I20260202 15:09:11.583988 12795 rpcz_store.cc:276] 0202 15:09:10.567765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.033559 (+465794us) service_pool.cc:225] Handling call
0202 15:09:11.059161 (+ 25602us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:11.135232 (+ 76071us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.583325 (+448093us) spinlock_profiling.cc:243] Waited 62 ms on lock 0x7b54002d3a54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:11.583523 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142656896}
W20260202 15:09:11.585341 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 44681.3us
I20260202 15:09:11.585983 12688 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:11.586112 15540 consensus_queue.cc:260] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.586858 15785 raft_consensus.cc:2804] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260202 15:09:11.587217 15785 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:11.587427 15540 raft_consensus.cc:3060] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:11.587525 15785 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Advancing to term 11
W20260202 15:09:11.588905 15811 consensus_meta.cc:287] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.525s user 0.010s sys 0.000s
I20260202 15:09:11.589334 15811 raft_consensus.cc:2749] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:11.590886 15569 raft_consensus.cc:2804] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader election won for term 9
I20260202 15:09:11.591383 15569 raft_consensus.cc:697] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:11.591490 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1875) took 1090 ms. Trace:
I20260202 15:09:11.591794 12726 rpcz_store.cc:276] 0202 15:09:10.501536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.737229 (+235693us) service_pool.cc:225] Handling call
0202 15:09:11.591324 (+854095us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b5400158e54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:11.591439 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":961778432}
I20260202 15:09:11.592267 15569 consensus_queue.cc:237] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:11.594779 15779 raft_consensus.cc:2749] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260202 15:09:11.600848 13355 raft_consensus.cc:3055] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:11.601186 13355 raft_consensus.cc:740] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:09:11.603592 15896 log.cc:927] Time spent T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20260202 15:09:11.604593 12650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1778) took 1050 ms. Trace:
W20260202 15:09:11.605181 15732 consensus_meta.cc:287] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.554s user 0.007s sys 0.000s
W20260202 15:09:11.605250 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.605633 15732 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.604938 12650 rpcz_store.cc:276] 0202 15:09:10.553757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.737776 (+184019us) service_pool.cc:225] Handling call
0202 15:09:10.740732 (+ 2956us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.541625 (+800893us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.604393 (+ 62768us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b5400213c54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:11.604541 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1822722816}
I20260202 15:09:11.306303 15808 raft_consensus.cc:515] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.608186 15808 leader_election.cc:290] T ee143496d46d4a45a2d24f0990e74f44 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:11.609215 12799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56714 (request call id 1441) took 1039 ms. Trace:
I20260202 15:09:11.609616 12799 rpcz_store.cc:276] 0202 15:09:10.569669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.038784 (+469115us) service_pool.cc:225] Handling call
0202 15:09:11.609077 (+570293us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260202 15:09:11.610515 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1874) took 1914 ms. Trace:
I20260202 15:09:11.558136 15417 raft_consensus.cc:515] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:11.610873 12724 rpcz_store.cc:276] 0202 15:09:09.695894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.117064 (+1421170us) service_pool.cc:225] Handling call
0202 15:09:11.179680 (+ 62616us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:11.342669 (+162989us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.610287 (+267618us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b5400018bd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55fa81 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786
0202 15:09:11.610461 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":310712576}
I20260202 15:09:11.612613 15417 leader_election.cc:290] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:11.612681 15414 raft_consensus.cc:3060] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:11.615135 12648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:58672 (request call id 27) took 1113 ms. Trace:
I20260202 15:09:11.615478 12648 rpcz_store.cc:276] 0202 15:09:10.502036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.513839 (+ 11803us) service_pool.cc:225] Handling call
0202 15:09:11.595769 (+1081930us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b54001a1dd4. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed5594d3 00007f24f251713a 00007f24ebab2934 00007f24ebab28c2 00007f24ebab27f1 00007f24ebab276c 00007f24ebab1a22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:11.615050 (+ 19281us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1825839872}
I20260202 15:09:11.581810 15712 consensus_queue.cc:260] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:11.620842 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.631461 12688 leader_election.cc:304] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
W20260202 15:09:11.637815 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 302472us
I20260202 15:09:11.626806 12544 catalog_manager.cc:5654] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 5, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194). New cstate: current_term: 5 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:09:11.651309 12544 catalog_manager.cc:5654] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 5 to 7, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195). New cstate: current_term: 7 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:09:11.653692 15706 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:11.654389 15706 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.654860 12644 raft_consensus.cc:3060] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:11.656280 15706 leader_election.cc:290] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:11.540593 15775 consensus_queue.cc:1059] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260202 15:09:11.654856 15588 raft_consensus.cc:493] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:11.660190 15588 raft_consensus.cc:515] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:11.660648 15574 consensus_queue.cc:1059] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260202 15:09:11.661294 15801 leader_election.cc:290] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:11.661981 15588 leader_election.cc:290] T 726983b999ef44c5a29a3f5d7461d92f P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:11.662045 15417 raft_consensus.cc:3055] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Stepping down as leader of term 8
I20260202 15:09:11.662778 15417 raft_consensus.cc:740] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:11.663723 15815 leader_election.cc:290] T 0f70fb0b2ffc4a51b06fe26a22355100 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:11.663950 15725 consensus_meta.cc:287] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.753s user 0.002s sys 0.007s
I20260202 15:09:11.663585 15417 consensus_queue.cc:260] T a3e391e37e424b1b8b2c63352ec8c198 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.664590 15833 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:11.664898 15588 raft_consensus.cc:2804] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260202 15:09:11.665052 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 134587us
I20260202 15:09:11.665060 15833 raft_consensus.cc:515] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.665875 12764 leader_election.cc:304] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:11.666647 15508 raft_consensus.cc:2749] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260202 15:09:11.667176 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 91411.9us
I20260202 15:09:11.669101 12689 leader_election.cc:304] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2
I20260202 15:09:11.670914 15588 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:11.671284 15588 raft_consensus.cc:3060] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:11.672415 12764 leader_election.cc:304] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:11.645381 15856 raft_consensus.cc:3055] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:11.673708 15813 raft_consensus.cc:2749] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:11.674533 15508 consensus_queue.cc:1048] T e4d8302d577042feaa44f63d596dc87d P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:09:11.674940 13328 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:11.675853 15508 raft_consensus.cc:2749] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260202 15:09:11.677373 15857 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.678268 12644 raft_consensus.cc:1275] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260202 15:09:11.679437 15578 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:11.673682 15856 raft_consensus.cc:740] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:11.567593 15821 raft_consensus.cc:515] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.650259 15779 consensus_queue.cc:1059] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:09:11.683225 15774 raft_consensus.cc:2749] T d3a3797f335d43dca87344d8954a1158 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260202 15:09:11.700223 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 48382.7us
I20260202 15:09:11.653142 12544 catalog_manager.cc:5654] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 3, leader changed from bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193) to 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196). New cstate: current_term: 3 leader_uuid: "66b8c715b9e6479e9218b08bd3156380" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } } }
I20260202 15:09:11.713990 12764 leader_election.cc:304] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:11.718930 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:11.714169 15804 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260202 15:09:11.724817 12615 leader_election.cc:304] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:11.741863 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.646459 15503 raft_consensus.cc:2749] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191) with higher term. Message: Invalid argument: T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8.
I20260202 15:09:11.751659 12644 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f8f84948a93c4dc984e69bfd1f44be23"
dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
mode: GRACEFUL
new_leader_uuid: "66b8c715b9e6479e9218b08bd3156380"
from {username='slave'} at 127.0.0.1:58672
I20260202 15:09:11.660671 15845 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:11.724594 15870 consensus_queue.cc:1059] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260202 15:09:11.771400 15588 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:11.773780 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.775135 15743 raft_consensus.cc:2804] T 488d33b86df249748c59fbf945f4e8b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election won for term 6
I20260202 15:09:11.757098 15732 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 10 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
W20260202 15:09:11.778987 15861 consensus_meta.cc:287] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380: Time spent flushing consensus metadata: real 0.681s user 0.007s sys 0.002s
W20260202 15:09:11.779085 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 100830us
I20260202 15:09:11.779685 13356 raft_consensus.cc:2468] T 5a776a09df954aa6ac5c756e8da6c56a P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 7.
I20260202 15:09:11.779935 15508 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:11.780115 12764 leader_election.cc:304] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:11.781682 12764 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:11.782996 15893 log.cc:927] Time spent T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.285s user 0.000s sys 0.001s
I20260202 15:09:11.783445 15712 raft_consensus.cc:3060] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:11.747080 15791 raft_consensus.cc:515] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:11.786243 15791 leader_election.cc:290] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:09:11.790951 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 46584.6us
W20260202 15:09:11.791087 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 172387us
I20260202 15:09:11.791917 12688 leader_election.cc:304] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:11.792392 12690 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters: bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:11.758522 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 251399us
I20260202 15:09:11.752206 12644 raft_consensus.cc:606] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Received request to transfer leadership to TS 66b8c715b9e6479e9218b08bd3156380
W20260202 15:09:11.761821 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 46528.7us
I20260202 15:09:11.762408 12726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed5f5e04d064d6e8d431aa2209cc141" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:11.795213 12644 raft_consensus.cc:614] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260202 15:09:11.762861 15898 log.cc:927] Time spent T d88e30bfa6e641749640dde238a2a8f8 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20260202 15:09:11.762979 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:11.570133 12718 rpcz_store.cc:276] 0202 15:09:10.567475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.235675 (+668200us) service_pool.cc:225] Handling call
0202 15:09:11.253138 (+ 17463us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.272400 (+ 19262us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:09:11.378817 (+106417us) log.cc:844] Serialized 14 byte log entry
0202 15:09:11.560743 (+181926us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:11.560799 (+ 56us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:09:11.569427 (+ 8628us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.569507 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:11.569627 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25736,"prepare.queue_time_us":200607,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":34333,"thread_start_us":55899,"threads_started":1}
I20260202 15:09:11.767233 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:11.766991 15856 consensus_queue.cc:260] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:11.768258 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.797722 12724 raft_consensus.cc:2410] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b8c715b9e6479e9218b08bd3156380 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260202 15:09:11.750912 12544 catalog_manager.cc:5654] T 0d12e74920dc474599bb20f8e977a6b4 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1eca64246cad41379cde4e2f9c3e25d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:09:11.793742 15821 leader_election.cc:290] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:11.821101 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:11.824575 12725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37626 (request call id 1876) took 1322 ms (client timeout 1725 ms). Trace:
W20260202 15:09:11.824924 12725 rpcz_store.cc:269] 0202 15:09:10.501938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.895052 (+393114us) service_pool.cc:225] Handling call
0202 15:09:11.820443 (+925391us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b5400157f54. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed562f7a 00007f24f2514647 00007f24ebaa4834 00007f24ebaa47c2 00007f24ebaa46f1 00007f24ebaa466c 00007f24ebaa3922 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:11.824524 (+ 4081us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1093867392}
I20260202 15:09:11.825786 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4798df17a1e4d18a594bb1b37b98bae" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:11.827711 13361 raft_consensus.cc:2393] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 8: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
W20260202 15:09:11.810987 15721 consensus_meta.cc:287] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2: Time spent flushing consensus metadata: real 0.587s user 0.005s sys 0.005s
I20260202 15:09:11.796975 15608 raft_consensus.cc:2804] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Leader election won for term 10
I20260202 15:09:11.796425 15813 raft_consensus.cc:2749] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260202 15:09:11.831930 13328 connection.cc:232] RPC call timeout handler was delayed by -1.6353454580000744s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
I20260202 15:09:11.836550 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:11.837059 12724 raft_consensus.cc:3060] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:11.836120 15911 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Leader election won for term 5
I20260202 15:09:11.832386 12718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "488d33b86df249748c59fbf945f4e8b3" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:11.840488 12796 raft_consensus.cc:2410] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260202 15:09:11.842041 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 50226.5us
I20260202 15:09:11.815088 15800 raft_consensus.cc:493] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:11.847437 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.818558 15503 raft_consensus.cc:2804] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260202 15:09:11.857915 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.839016 15911 raft_consensus.cc:697] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Leader. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:11.859031 12544 catalog_manager.cc:5654] T d3a3797f335d43dca87344d8954a1158 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:09:11.794126 15785 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.839102 15871 raft_consensus.cc:493] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:11.844182 15725 raft_consensus.cc:3055] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Stepping down as leader of term 6
I20260202 15:09:11.839756 15588 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:11.838385 15876 consensus_queue.cc:1059] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.016s
W20260202 15:09:11.871806 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 90232.6us
I20260202 15:09:11.856745 15503 raft_consensus.cc:493] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:11.865222 15837 raft_consensus.cc:2804] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260202 15:09:11.863292 15870 raft_consensus.cc:2804] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:09:11.838953 15880 consensus_queue.cc:1059] T d30f8f27ce00483c85fb1758a4f81f03 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
W20260202 15:09:11.893213 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 100903us
W20260202 15:09:11.902716 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.862306 15608 raft_consensus.cc:697] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:11.889242 12544 catalog_manager.cc:5654] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:09:11.932973 13357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:11.967799 11451 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260202 15:09:11.877929 15725 raft_consensus.cc:740] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:11.848789 15673 consensus_queue.cc:1048] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260202 15:09:11.867851 15721 consensus_queue.cc:1048] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.043s
I20260202 15:09:11.973728 15698 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:11.980649 15698 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:11.981925 11451 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260202 15:09:11.984381 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 156722us
I20260202 15:09:11.979658 15608 consensus_queue.cc:237] T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:11.903792 15503 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:11.904778 15837 raft_consensus.cc:493] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:11.889910 15821 consensus_queue.cc:1048] T a739ed9c340d4dd9935861456bc3f3d3 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20260202 15:09:11.868902 15800 raft_consensus.cc:515] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:12.012321 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 183554us
W20260202 15:09:12.017212 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 89765.8us
W20260202 15:09:12.020008 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.012130 11451 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260202 15:09:12.046862 12689 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:11.921572 15876 consensus_queue.cc:1243] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer 6921687ed3814cccb2bd9c606875b9f0 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:09:12.079283 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1728) took 2481 ms. Trace:
W20260202 15:09:12.084954 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 45530.6us
I20260202 15:09:12.084823 12721 rpcz_store.cc:276] 0202 15:09:09.598022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.063116 (+1465094us) service_pool.cc:225] Handling call
0202 15:09:11.063511 (+ 395us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:11.065252 (+ 1741us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:11.065279 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:11.065677 (+ 398us) log.cc:844] Serialized 26 byte log entry
0202 15:09:11.187581 (+121904us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:11.187611 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.187663 (+ 52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:12.079087 (+891424us) raft_consensus.cc:1685] finished
0202 15:09:12.079110 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:12.079228 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":104804,"threads_started":1,"wal-append.queue_time_us":469704}
I20260202 15:09:12.083428 15861 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.087313 12689 leader_election.cc:304] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:12.037734 12544 catalog_manager.cc:5654] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "6921687ed3814cccb2bd9c606875b9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } }
I20260202 15:09:12.012712 15837 raft_consensus.cc:3060] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Advancing to term 8
W20260202 15:09:12.098093 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.012549 15789 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:12.127347 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 261372us
I20260202 15:09:12.038553 15857 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:12.134948 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:12.012140 15799 consensus_queue.cc:1048] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.126s
I20260202 15:09:11.895988 15871 raft_consensus.cc:515] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.037392 15870 raft_consensus.cc:697] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 LEADER]: Becoming Leader. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:12.143867 12615 leader_election.cc:304] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:12.091215 15721 raft_consensus.cc:493] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.096684 15880 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.091467 15414 raft_consensus.cc:493] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.085621 15813 raft_consensus.cc:493] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1eca64246cad41379cde4e2f9c3e25d2)
W20260202 15:09:12.150980 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 66336.9us
I20260202 15:09:12.092022 15698 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:12.143532 12544 catalog_manager.cc:5654] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "bbf140958c3b4f339c512a9dc4be2dab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260202 15:09:12.162174 12721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:12.163556 12725 raft_consensus.cc:1275] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260202 15:09:12.154855 13326 leader_election.cc:304] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:12.163964 12725 pending_rounds.cc:85] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Aborting all ops after (but not including) 7
I20260202 15:09:12.169313 12725 pending_rounds.cc:107] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260202 15:09:12.169598 12725 raft_consensus.cc:2889] T a50a506e17f747e0a940bc3c982336c6 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:12.124418 15861 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:12.184556 15891 log.cc:927] Time spent T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260202 15:09:12.166193 15789 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.188416 15789 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.188551 15508 raft_consensus.cc:2749] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:12.195027 15785 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 11 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:12.195644 15897 log.cc:927] Time spent T da327288552c40f59790d9001c7f7e5d P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.207s user 0.000s sys 0.000s
W20260202 15:09:12.207770 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.209142 12724 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 8.
I20260202 15:09:12.212409 12791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56734 (request call id 1670) took 1650 ms. Trace:
I20260202 15:09:12.156548 15540 raft_consensus.cc:2749] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:12.215441 12791 rpcz_store.cc:276] 0202 15:09:10.562235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.000489 (+438254us) service_pool.cc:225] Handling call
0202 15:09:11.019586 (+ 19097us) raft_consensus.cc:1471] Updating replica for 2 ops
0202 15:09:11.061651 (+ 42065us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:11.061683 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:11.062294 (+ 611us) log.cc:844] Serialized 52 byte log entry
0202 15:09:11.144215 (+ 81921us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:11.179936 (+ 35721us) log.cc:844] Serialized 14 byte log entry
0202 15:09:11.187244 (+ 7308us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.187302 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:12.196809 (+1009507us) raft_consensus.cc:1685] finished
0202 15:09:12.196827 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:12.212344 (+ 15517us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25721,"prepare.queue_time_us":35593,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":36162,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":11738,"thread_start_us":81163,"threads_started":1,"wal-append.queue_time_us":794045}
I20260202 15:09:12.232690 15673 raft_consensus.cc:2749] T a3e391e37e424b1b8b2c63352ec8c198 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:12.233582 13356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
W20260202 15:09:12.234783 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.238240 13328 leader_election.cc:304] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
W20260202 15:09:12.239317 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 221142us
W20260202 15:09:12.239934 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.239840 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.119393 15911 consensus_queue.cc:237] T 65f098e59c0240a3a65cbd15dc358b64 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:12.245252 12766 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.244074 12724 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8.
W20260202 15:09:12.248652 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 104859us
I20260202 15:09:12.143528 15725 consensus_queue.cc:260] T 435e192b63f348a58c00c3ae9e598ea9 P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.161533 15721 raft_consensus.cc:515] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.172163 15880 raft_consensus.cc:515] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.233723 15857 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:12.254854 12765 leader_election.cc:304] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:12.248577 15673 raft_consensus.cc:2804] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:09:12.244691 15929 raft_consensus.cc:2749] T 7d28e4999b504df4a53610187731416d P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260202 15:09:12.262743 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:12.263444 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 131750us
I20260202 15:09:12.219542 15813 raft_consensus.cc:515] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:12.263922 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 119403us
I20260202 15:09:12.265451 12615 leader_election.cc:304] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:12.258630 15673 raft_consensus.cc:697] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [term 8 LEADER]: Becoming Leader. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:12.268985 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:12.270191 12724 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.269901 15673 consensus_queue.cc:237] T 6f9321e2251943229a0b7dbdc0577c6b P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.271401 12724 raft_consensus.cc:1275] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Refusing update from remote peer 66b8c715b9e6479e9218b08bd3156380: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:09:12.242470 15414 raft_consensus.cc:515] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.277379 12723 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.258816 15789 raft_consensus.cc:2804] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:12.263681 15540 raft_consensus.cc:493] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260202 15:09:12.275009 15663 consensus_meta.cc:287] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab: Time spent flushing consensus metadata: real 0.640s user 0.008s sys 0.001s
I20260202 15:09:12.293617 13326 leader_election.cc:304] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:12.294543 12721 raft_consensus.cc:3060] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Advancing to term 10
I20260202 15:09:12.285441 15791 raft_consensus.cc:2804] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260202 15:09:12.300324 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:12.300727 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.261096 15857 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.268098 15698 consensus_queue.cc:1059] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
W20260202 15:09:12.309347 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 51067us
I20260202 15:09:12.286734 15789 raft_consensus.cc:493] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:12.312458 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.294837 15845 raft_consensus.cc:493] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:12.323830 12650 raft_consensus.cc:1275] T dbc759e31af04c60b10baa9dce36e825 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260202 15:09:12.312989 12642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:12.324950 12644 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:12.292126 15876 consensus_queue.cc:1048] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260202 15:09:12.299713 15791 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
W20260202 15:09:12.329603 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 417637us
I20260202 15:09:12.306895 15800 leader_election.cc:290] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:12.315824 15774 raft_consensus.cc:2749] T 4d879eebd1924ad8b6f65647f4e73362 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:12.333988 12764 leader_election.cc:304] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:12.318925 15789 raft_consensus.cc:3060] T 3a02578a734446c387f55d747a3f3bd0 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:12.298677 15663 raft_consensus.cc:515] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.298857 15540 raft_consensus.cc:515] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.340240 12724 raft_consensus.cc:1240] T 11a10c0c64914444a6659e661207c082 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 6. Current term is 7. Ops: []
I20260202 15:09:12.343123 12721 raft_consensus.cc:2468] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b8c715b9e6479e9218b08bd3156380 in term 10.
I20260202 15:09:12.335602 15791 raft_consensus.cc:3060] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
I20260202 15:09:12.354907 15917 raft_consensus.cc:2749] T 9592591cd602455aa52fdc1d84e48b63 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260202 15:09:12.356606 13359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44934 (request call id 750) took 1407 ms. Trace:
I20260202 15:09:12.362411 15861 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
W20260202 15:09:12.367065 15916 log.cc:927] Time spent T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20260202 15:09:12.367587 15926 raft_consensus.cc:493] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 66b8c715b9e6479e9218b08bd3156380)
I20260202 15:09:12.381955 12642 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Advancing to term 11
W20260202 15:09:12.368786 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 85680.3us
W20260202 15:09:12.369187 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:12.371157 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.368081 12615 leader_election.cc:304] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:12.383965 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 44899.4us
I20260202 15:09:12.384585 12688 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab; no voters:
I20260202 15:09:12.313477 15870 consensus_queue.cc:237] T 9fae829b1c574bc795c5a424036329b3 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.387094 12615 leader_election.cc:304] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:12.387897 12764 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:12.389503 15921 raft_consensus.cc:2764] T 5a776a09df954aa6ac5c756e8da6c56a P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260202 15:09:12.389976 12615 leader_election.cc:304] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:12.340137 15569 consensus_queue.cc:1048] T 87df3a9146bb41e1ac7468e17d2d9498 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260202 15:09:12.391183 15608 consensus_queue.cc:1048] T dbc759e31af04c60b10baa9dce36e825 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:09:12.391534 15928 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:12.391918 15928 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:12.392205 15928 raft_consensus.cc:3060] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Advancing to term 9
W20260202 15:09:12.392266 12763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.028s sys 0.052s
W20260202 15:09:12.392607 12763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s user 0.028s sys 0.052s
W20260202 15:09:12.392824 12763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 836559us
I20260202 15:09:12.393404 15910 raft_consensus.cc:3060] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Advancing to term 8
W20260202 15:09:12.394299 15923 log.cc:927] Time spent T 21bc61b2f0f64d0b80e5d98615b1d6b6 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260202 15:09:12.371887 13359 rpcz_store.cc:276] 0202 15:09:10.951572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.191873 (+240301us) service_pool.cc:225] Handling call
0202 15:09:11.196473 (+ 4600us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:11.232118 (+ 35645us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:11.232158 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:11.264545 (+ 32387us) log.cc:844] Serialized 26 byte log entry
0202 15:09:11.794388 (+529843us) raft_consensus.cc:1645] Marking committed up to 6
0202 15:09:11.794430 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.794489 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:12.356427 (+561938us) raft_consensus.cc:1685] finished
0202 15:09:12.356444 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:12.356550 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16677,"raft.queue_time_us":37567,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":8863,"thread_start_us":480165,"threads_started":1,"wal-append.queue_time_us":1047938}
I20260202 15:09:12.396764 12724 raft_consensus.cc:1240] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 11 FOLLOWER]: Rejecting Update request from peer 66b8c715b9e6479e9218b08bd3156380 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260202 15:09:12.398910 13362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50a506e17f747e0a940bc3c982336c6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:12.400467 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 87992.8us
I20260202 15:09:12.404752 13360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:12.406739 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:12.400282 15914 log.cc:927] Time spent T ed220e8ce3ce45ff8b531267a15de19e P 1eca64246cad41379cde4e2f9c3e25d2: Append to log took a long time: real 0.245s user 0.001s sys 0.000s
I20260202 15:09:12.340525 15845 raft_consensus.cc:515] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.421110 13360 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 9.
I20260202 15:09:12.413364 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:12.415331 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.423952 12644 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P bbf140958c3b4f339c512a9dc4be2dab [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 10: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:12.411120 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:12.419312 12645 raft_consensus.cc:1240] T da327288552c40f59790d9001c7f7e5d P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:12.402027 15800 raft_consensus.cc:3060] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:12.427816 13354 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 6: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:12.434275 13359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:12.434916 13359 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:12.441691 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:12.416657 15721 leader_election.cc:290] T 435e192b63f348a58c00c3ae9e598ea9 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:12.444164 13361 raft_consensus.cc:2393] T ee143496d46d4a45a2d24f0990e74f44 P 66b8c715b9e6479e9218b08bd3156380 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 8: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:12.450791 13354 raft_consensus.cc:2393] T dc5a4c8371434c48926f011a134d3bd4 P 66b8c715b9e6479e9218b08bd3156380 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 10: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:12.451905 12689 leader_election.cc:304] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:12.417824 15880 leader_election.cc:290] T 4687c40305f64dd2960fcbb336f93395 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:12.441860 15503 raft_consensus.cc:515] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } attrs { replace: true } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.455430 13326 leader_election.cc:304] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:12.456051 15907 consensus_queue.cc:1059] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260202 15:09:12.412370 15926 raft_consensus.cc:515] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.431725 15813 leader_election.cc:290] T fe02793fdaa448caa739180083258aae P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:12.463369 15813 raft_consensus.cc:2804] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader election won for term 8
I20260202 15:09:12.463944 15813 raft_consensus.cc:697] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [term 8 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:12.464260 15926 leader_election.cc:290] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.465351 15540 leader_election.cc:290] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.464911 15813 consensus_queue.cc:237] T 83fc6cc808d9450788aeecc86caa94bd P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.474717 15845 leader_election.cc:290] T 7d28e4999b504df4a53610187731416d P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.476318 15926 consensus_queue.cc:1059] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260202 15:09:12.476719 15663 leader_election.cc:290] T 69e93e86b75b464fa4c55fdeb141acea P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 4 election: Requested vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.477514 15926 raft_consensus.cc:2749] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260202 15:09:12.435524 15414 leader_election.cc:290] T dc5a4c8371434c48926f011a134d3bd4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.480052 12644 raft_consensus.cc:1240] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:12.479350 12645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.481544 12645 raft_consensus.cc:2393] T 965bcb05a43147b7be53449d0fe46509 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 7: Already voted for candidate bbf140958c3b4f339c512a9dc4be2dab in this term.
I20260202 15:09:12.435738 15861 consensus_queue.cc:1059] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260202 15:09:12.493175 15775 raft_consensus.cc:2749] T f8f84948a93c4dc984e69bfd1f44be23 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:12.500622 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fc895893bbe4569bb328cfc3513de38" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:12.502060 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 112165us
I20260202 15:09:12.502332 12647 raft_consensus.cc:1275] T 01f20099bc6844498a65c943dde6b1ff P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260202 15:09:12.505420 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fae829b1c574bc795c5a424036329b3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.506026 12795 raft_consensus.cc:2376] T 9fae829b1c574bc795c5a424036329b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8. Re-sending same reply.
I20260202 15:09:12.483966 15857 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:12.483593 15721 raft_consensus.cc:493] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.490566 15574 raft_consensus.cc:493] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.507666 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d88e30bfa6e641749640dde238a2a8f8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.508493 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.504878 12798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.512952 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.514284 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc29e67ee294975abcdfc9e53f8b6e1" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.510902 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.490818 15608 raft_consensus.cc:493] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bbf140958c3b4f339c512a9dc4be2dab)
I20260202 15:09:12.521157 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30f8f27ce00483c85fb1758a4f81f03" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:12.521097 12799 raft_consensus.cc:1275] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260202 15:09:12.498234 15910 raft_consensus.cc:2749] T 9fae829b1c574bc795c5a424036329b3 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:12.522680 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc759e31af04c60b10baa9dce36e825" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.530081 12791 raft_consensus.cc:2393] T ebc29e67ee294975abcdfc9e53f8b6e1 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:12.513945 12615 leader_election.cc:304] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380
I20260202 15:09:12.509929 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a739ed9c340d4dd9935861456bc3f3d3" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.503279 15861 consensus_queue.cc:1243] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer 6921687ed3814cccb2bd9c606875b9f0 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260202 15:09:12.497447 15698 raft_consensus.cc:3055] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:12.547598 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac5476ea30a46548db9ede417d946e6" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.551573 12644 raft_consensus.cc:2410] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260202 15:09:12.551692 13328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 97143us
I20260202 15:09:12.551574 12800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f72d171c95a149d890432428ecf80739" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:12.494205 15880 consensus_queue.cc:1059] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260202 15:09:12.553682 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a02578a734446c387f55d747a3f3bd0" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.554678 12647 raft_consensus.cc:1240] T f8f84948a93c4dc984e69bfd1f44be23 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: []
I20260202 15:09:12.555191 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.554512 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:12.548318 12689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 94792.8us
I20260202 15:09:12.558983 12793 raft_consensus.cc:2393] T d30f8f27ce00483c85fb1758a4f81f03 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 7: Already voted for candidate 1eca64246cad41379cde4e2f9c3e25d2 in this term.
I20260202 15:09:12.560129 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726983b999ef44c5a29a3f5d7461d92f" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.560940 12644 raft_consensus.cc:2410] T 726983b999ef44c5a29a3f5d7461d92f P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
W20260202 15:09:12.560865 12690 connection.cc:232] RPC call timeout handler was delayed by -1.5490627929998482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1791s for a response
I20260202 15:09:12.564000 12643 raft_consensus.cc:2468] T 3a02578a734446c387f55d747a3f3bd0 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8.
I20260202 15:09:12.564967 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.565461 12643 raft_consensus.cc:2468] T 4687c40305f64dd2960fcbb336f93395 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 6.
I20260202 15:09:12.566368 12795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f9321e2251943229a0b7dbdc0577c6b" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.565460 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:12.565951 12649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.567615 12792 raft_consensus.cc:1240] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Rejecting Update request from peer 6921687ed3814cccb2bd9c606875b9f0 for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20260202 15:09:12.567806 12649 raft_consensus.cc:2468] T d2476071e1d24aa6a1342c8d81699cef P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 9.
I20260202 15:09:12.568461 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee143496d46d4a45a2d24f0990e74f44" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.541661 15870 consensus_queue.cc:1059] T 726983b999ef44c5a29a3f5d7461d92f P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260202 15:09:12.509227 15508 consensus_queue.cc:1048] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260202 15:09:12.552623 15698 raft_consensus.cc:740] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
I20260202 15:09:12.566975 12795 raft_consensus.cc:2468] T 6f9321e2251943229a0b7dbdc0577c6b P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:12.506315 15917 consensus_queue.cc:1059] T 0d12e74920dc474599bb20f8e977a6b4 P 1eca64246cad41379cde4e2f9c3e25d2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260202 15:09:12.546077 15926 consensus_queue.cc:1048] T 01f20099bc6844498a65c943dde6b1ff P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.025s
I20260202 15:09:12.539640 15721 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.571250 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:12.557750 15503 leader_election.cc:290] T 015f643434d84eac8a3774b663eb4220 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 election: Requested vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.559387 15821 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.572980 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b34ca86c2f4e9b831f13ebf9cd3aa8" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:12.569393 12641 pending_rounds.cc:85] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 4
I20260202 15:09:12.545889 15574 raft_consensus.cc:515] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.577292 15833 leader_election.cc:290] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.600291 12641 pending_rounds.cc:107] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260202 15:09:12.581905 12649 raft_consensus.cc:1275] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260202 15:09:12.583892 12646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.583902 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.596740 15706 raft_consensus.cc:3055] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Stepping down as leader of term 8
I20260202 15:09:12.595558 15721 leader_election.cc:290] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:12.611783 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:12.618856 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f098e59c0240a3a65cbd15dc358b64" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:12.593665 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.594578 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bc61b2f0f64d0b80e5d98615b1d6b6" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.599009 12689 leader_election.cc:304] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 66b8c715b9e6479e9218b08bd3156380, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:12.599174 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 85606.6us
I20260202 15:09:12.624759 12642 raft_consensus.cc:2468] T 9335cdc844e44d608b9cdd8369f9afb1 P bbf140958c3b4f339c512a9dc4be2dab [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 11.
I20260202 15:09:12.578841 15861 raft_consensus.cc:3055] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 10 LEADER]: Stepping down as leader of term 10
I20260202 15:09:12.563921 15608 raft_consensus.cc:515] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } attrs { replace: true } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.584571 15800 raft_consensus.cc:2749] T f72d171c95a149d890432428ecf80739 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:12.623667 13354 raft_consensus.cc:2468] T 65f098e59c0240a3a65cbd15dc358b64 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 5.
I20260202 15:09:12.631352 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:12.625223 12641 raft_consensus.cc:2889] T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260202 15:09:12.612138 13355 consensus_queue.cc:260] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.624094 12793 raft_consensus.cc:2468] T 83fc6cc808d9450788aeecc86caa94bd P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 8.
I20260202 15:09:12.640050 12689 leader_election.cc:304] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0; no voters:
I20260202 15:09:12.624639 12646 raft_consensus.cc:2410] T 435e192b63f348a58c00c3ae9e598ea9 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260202 15:09:12.637261 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 65740.4us
I20260202 15:09:12.622588 12764 leader_election.cc:304] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:12.642498 12764 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:12.643483 12643 raft_consensus.cc:1275] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260202 15:09:12.643878 13355 raft_consensus.cc:3060] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Advancing to term 7
W20260202 15:09:12.646783 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.647109 15943 raft_consensus.cc:2804] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader election won for term 7
I20260202 15:09:12.647423 15721 raft_consensus.cc:2749] T a50a506e17f747e0a940bc3c982336c6 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260202 15:09:12.647598 15943 raft_consensus.cc:697] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [term 7 LEADER]: Becoming Leader. State: Replica: bbf140958c3b4f339c512a9dc4be2dab, State: Running, Role: LEADER
I20260202 15:09:12.646864 15663 consensus_queue.cc:1048] T 78ef01280730461580740297403fe090 P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260202 15:09:12.648372 15944 raft_consensus.cc:493] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.648509 15943 consensus_queue.cc:237] T 5a776a09df954aa6ac5c756e8da6c56a P bbf140958c3b4f339c512a9dc4be2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.648836 15944 raft_consensus.cc:515] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.649575 15808 raft_consensus.cc:3055] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Stepping down as leader of term 5
I20260202 15:09:12.649976 15808 raft_consensus.cc:740] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:12.650413 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1743) took 1770 ms. Trace:
I20260202 15:09:12.641607 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37626 (request call id 1854) took 3241 ms. Trace:
I20260202 15:09:12.651654 12764 leader_election.cc:304] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:12.640072 12649 pending_rounds.cc:85] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: Aborting all ops after (but not including) 3
I20260202 15:09:12.652768 12649 pending_rounds.cc:107] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260202 15:09:12.653131 12649 raft_consensus.cc:2889] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260202 15:09:12.655053 12615 connection.cc:232] RPC call timeout handler was delayed by -2.0474946740000632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
I20260202 15:09:12.648833 15926 raft_consensus.cc:2804] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260202 15:09:12.652732 12764 leader_election.cc:304] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:12.656251 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "015f643434d84eac8a3774b663eb4220" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380"
I20260202 15:09:12.656370 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00b1a15d2d1246e1abe52586ddf512ef" candidate_uuid: "66b8c715b9e6479e9218b08bd3156380" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.656955 13358 raft_consensus.cc:3060] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 7 LEARNER]: Advancing to term 8
I20260202 15:09:12.641336 15910 raft_consensus.cc:493] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:12.650753 12719 rpcz_store.cc:276] 0202 15:09:10.892289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.504964 (+612675us) service_pool.cc:225] Handling call
0202 15:09:11.618269 (+113305us) raft_consensus.cc:1471] Updating replica for 1 ops
0202 15:09:11.751325 (+133056us) raft_consensus.cc:1534] Early marking committed up to index 0
0202 15:09:11.751354 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0202 15:09:11.771593 (+ 20239us) log.cc:844] Serialized 26 byte log entry
0202 15:09:11.792420 (+ 20827us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b180030a840. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4b0ae2 00007f24ed0f85ab 00007f24ed0fe0c3 00007f24ed0fe471 00007f24ed51a73f
0202 15:09:11.888208 (+ 95788us) raft_consensus.cc:1645] Marking committed up to 3
0202 15:09:11.888256 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:11.888330 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:12.395539 (+507209us) raft_consensus.cc:1685] finished
0202 15:09:12.395555 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:12.650271 (+254716us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x7b5400158e50. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed55e52d 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22 00007f24eb89e34c 00007f24eb89d786 00007f24eb89d12f
0202 15:09:12.650365 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":113857,"raft.queue_time_us":97352,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":9945,"spinlock_wait_cycles":242597888,"thread_start_us":95618,"threads_started":1,"wal-append.queue_time_us":473356}
I20260202 15:09:12.667261 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc5a4c8371434c48926f011a134d3bd4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:12.613135 15572 raft_consensus.cc:3060] T 488d33b86df249748c59fbf945f4e8b3 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:12.666266 12647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
W20260202 15:09:12.675911 12615 leader_election.cc:336] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Timed out: RequestConsensusVote RPC to 127.11.46.194:44367 timed out after 1.552s (SENT)
I20260202 15:09:12.651623 12720 rpcz_store.cc:276] 0202 15:09:09.386750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:10.929515 (+1542765us) service_pool.cc:225] Handling call
0202 15:09:10.988046 (+ 58531us) raft_consensus.cc:1471] Updating replica for 3 ops
0202 15:09:11.199594 (+211548us) raft_consensus.cc:1534] Early marking committed up to index 5
0202 15:09:11.524040 (+324446us) log.cc:844] Serialized 14 byte log entry
0202 15:09:12.166822 (+642782us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0202 15:09:12.613585 (+446763us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b2c00010828. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ebe5d9b1 00007f24ebe6cfe2 00007f24ed59a851 00007f24ed55f0ba 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c 00007f24eba9ee22
0202 15:09:12.614261 (+ 676us) log.cc:844] Serialized 590 byte log entry
0202 15:09:12.614394 (+ 133us) raft_consensus.cc:1645] Marking committed up to 5
0202 15:09:12.614423 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:12.614488 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0202 15:09:12.624485 (+ 9997us) raft_consensus.cc:1685] finished
0202 15:09:12.628371 (+ 3886us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:12.628507 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47231,"prepare.queue_time_us":263146,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":11086,"raft.queue_time_us":83350,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":85690,"spinlock_wait_cycles":8701824,"thread_start_us":815879,"threads_started":3,"wal-append.queue_time_us":610502}
I20260202 15:09:12.614589 15806 raft_consensus.cc:2749] T 015f643434d84eac8a3774b663eb4220 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:12.677052 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c6861781ee34db9b79870494af1cced" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:12.641475 15937 log.cc:927] Time spent T 323d926990494ad1aafb6aa0282d9f68 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260202 15:09:12.626000 15861 raft_consensus.cc:740] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
I20260202 15:09:12.754951 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "323d926990494ad1aafb6aa0282d9f68" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.630831 15706 raft_consensus.cc:740] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1eca64246cad41379cde4e2f9c3e25d2, State: Running, Role: LEADER
I20260202 15:09:12.706125 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37612 (request call id 1729) took 3107 ms. Trace:
W20260202 15:09:12.700002 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 244654us
W20260202 15:09:12.745992 12765 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 68251.5us
I20260202 15:09:12.631258 15833 raft_consensus.cc:3060] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:12.666993 15574 leader_election.cc:290] T 1c6861781ee34db9b79870494af1cced P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367)
I20260202 15:09:12.748575 12647 raft_consensus.cc:2468] T 30df58c1134a49e18c29202abd31da41 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921687ed3814cccb2bd9c606875b9f0 in term 6.
I20260202 15:09:12.761620 15778 raft_consensus.cc:2804] T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260202 15:09:12.650717 15808 consensus_queue.cc:260] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.762068 15676 raft_consensus.cc:2804] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260202 15:09:12.762480 15676 raft_consensus.cc:493] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:12.762785 15808 raft_consensus.cc:3060] T e9b34ca86c2f4e9b831f13ebf9cd3aa8 P 1eca64246cad41379cde4e2f9c3e25d2 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:12.769059 15861 consensus_queue.cc:260] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:12.749625 12764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 45762.8us
I20260202 15:09:12.770361 15861 raft_consensus.cc:3060] T 9335cdc844e44d608b9cdd8369f9afb1 P 66b8c715b9e6479e9218b08bd3156380 [term 10 FOLLOWER]: Advancing to term 11
I20260202 15:09:12.668649 15608 leader_election.cc:290] T 78ef01280730461580740297403fe090 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.655349 15821 raft_consensus.cc:515] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.772351 12795 raft_consensus.cc:1275] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260202 15:09:12.773280 15821 leader_election.cc:290] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.774981 13326 leader_election.cc:304] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:12.655871 15944 leader_election.cc:290] T 30df58c1134a49e18c29202abd31da41 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bbf140958c3b4f339c512a9dc4be2dab (127.11.46.193:40057), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:12.618525 15791 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.611510 15880 consensus_queue.cc:1048] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.073s
I20260202 15:09:12.640738 15935 raft_consensus.cc:493] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.754762 12644 raft_consensus.cc:1275] T 1c6861781ee34db9b79870494af1cced P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260202 15:09:12.749934 12648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab" is_pre_election: true
I20260202 15:09:12.656034 15926 raft_consensus.cc:493] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:12.666541 15779 raft_consensus.cc:2749] T 78ef01280730461580740297403fe090 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260202 15:09:12.718958 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.759476 12690 leader_election.cc:304] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:12.760247 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
W20260202 15:09:12.761327 12614 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.769497 15833 raft_consensus.cc:2749] T 965bcb05a43147b7be53449d0fe46509 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260202 15:09:12.709532 12615 leader_election.cc:304] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 6921687ed3814cccb2bd9c606875b9f0
I20260202 15:09:12.752406 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "965bcb05a43147b7be53449d0fe46509" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:12.788573 13358 raft_consensus.cc:2468] T 015f643434d84eac8a3774b663eb4220 P 66b8c715b9e6479e9218b08bd3156380 [term 8 LEARNER]: Leader election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 8.
I20260202 15:09:12.749775 15721 raft_consensus.cc:2804] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260202 15:09:12.816551 13354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78ef01280730461580740297403fe090" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:12.817237 13354 raft_consensus.cc:2393] T 78ef01280730461580740297403fe090 P 66b8c715b9e6479e9218b08bd3156380 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6921687ed3814cccb2bd9c606875b9f0 in current term 6: Already voted for candidate 66b8c715b9e6479e9218b08bd3156380 in this term.
I20260202 15:09:12.757416 15800 raft_consensus.cc:493] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.797948 13326 leader_election.cc:304] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 6921687ed3814cccb2bd9c606875b9f0, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:12.820474 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 44196.3us
W20260202 15:09:12.804333 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.758033 12717 rpcz_store.cc:276] 0202 15:09:09.598285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.097481 (+1499196us) service_pool.cc:225] Handling call
0202 15:09:11.135660 (+ 38179us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.141894 (+ 6234us) raft_consensus.cc:1534] Early marking committed up to index 9
0202 15:09:11.261941 (+120047us) log.cc:844] Serialized 14 byte log entry
0202 15:09:12.134949 (+873008us) log.cc:844] Serialized 14 byte log entry
0202 15:09:12.579931 (+444982us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b18003bf020. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24f3305c91 00007f24f3312671 00007f24ea493c67 00007f24f327c709 00007f24ea4b057b 00007f24ea4ae81f 00007f24ea4ae335 00007f24ed556ce0 00007f24ed569396 00007f24ed579f07 00007f24ed579e88
0202 15:09:12.705876 (+125945us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0202 15:09:12.705908 (+ 32us) raft_consensus.cc:1645] Marking committed up to 9
0202 15:09:12.705944 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:12.706010 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0202 15:09:12.706088 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98397,"prepare.queue_time_us":240708,"prepare.run_cpu_time_us":6827,"prepare.run_wall_time_us":46411,"raft.queue_time_us":178704,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":9697,"spinlock_wait_cycles":8626560,"thread_start_us":944411,"threads_started":4,"wal-append.queue_time_us":1110161}
W20260202 15:09:12.808712 12613 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.821928 13358 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
I20260202 15:09:12.809995 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d28e4999b504df4a53610187731416d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.822417 13358 raft_consensus.cc:2468] T 3b57a31e08c1420ab954d13d66af9297 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eca64246cad41379cde4e2f9c3e25d2 in term 9.
I20260202 15:09:12.813112 13355 raft_consensus.cc:1275] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Refusing update from remote peer 6921687ed3814cccb2bd9c606875b9f0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260202 15:09:12.813652 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9335cdc844e44d608b9cdd8369f9afb1" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.823096 13355 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44940 (request call id 932) took 1896 ms. Trace:
I20260202 15:09:12.823311 13355 rpcz_store.cc:276] 0202 15:09:10.927045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0202 15:09:11.192656 (+265611us) service_pool.cc:225] Handling call
0202 15:09:11.264315 (+ 71659us) raft_consensus.cc:1471] Updating replica for 0 ops
0202 15:09:11.600760 (+336445us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b540045de20. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4d6df0 00007f24ed560e40 00007f24ed561eb1 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2 00007f24eba9fbf1 00007f24eba9fb6c
0202 15:09:12.598675 (+997915us) spinlock_profiling.cc:243] Waited 79.1 ms on lock 0x7b540045de20. stack: 00007f24ea47f373 00007f24ea47f270 00007f24e9dd1574 00007f24f3a8d6b4 00007f24f3a8d649 00007f24ed4c85a1 00007f24ed4dd09f 00007f24ed555be0 00007f24ed55745e 00007f24ed5617c9 00007f24ed562096 00007f24ed55ea0f 00007f24ed55e4be 00007f24f2512f49 00007f24eba9fd34 00007f24eba9fcc2
0202 15:09:12.822893 (+224218us) raft_consensus.cc:1697] Filling consensus response to leader.
0202 15:09:12.823052 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":731344512}
I20260202 15:09:12.611660 15698 consensus_queue.cc:260] T da327288552c40f59790d9001c7f7e5d P 6921687ed3814cccb2bd9c606875b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.762861 15676 raft_consensus.cc:3060] T d2476071e1d24aa6a1342c8d81699cef P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Advancing to term 10
I20260202 15:09:12.780779 13361 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30df58c1134a49e18c29202abd31da41" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "66b8c715b9e6479e9218b08bd3156380" is_pre_election: true
W20260202 15:09:12.811581 12691 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.815423 12797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f70fb0b2ffc4a51b06fe26a22355100" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:12.824960 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d879eebd1924ad8b6f65647f4e73362" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.834303 12797 raft_consensus.cc:3060] T 0f70fb0b2ffc4a51b06fe26a22355100 P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Advancing to term 9
I20260202 15:09:12.824970 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af412f1b23854fdea68917d1df084a35" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:12.830214 12765 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.831256 12720 raft_consensus.cc:2393] T 9335cdc844e44d608b9cdd8369f9afb1 P 6921687ed3814cccb2bd9c606875b9f0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf140958c3b4f339c512a9dc4be2dab in current term 11: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
I20260202 15:09:12.792788 15876 raft_consensus.cc:2749] T 87df3a9146bb41e1ac7468e17d2d9498 P 66b8c715b9e6479e9218b08bd3156380 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260202 15:09:12.833652 12725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.837006 12764 leader_election.cc:304] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380; no voters:
I20260202 15:09:12.838456 12792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2476071e1d24aa6a1342c8d81699cef" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.842417 12791 raft_consensus.cc:1240] T 6fc895893bbe4569bb328cfc3513de38 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 LEADER]: Rejecting Update request from peer bbf140958c3b4f339c512a9dc4be2dab for earlier term 5. Current term is 6. Ops: []
I20260202 15:09:12.812209 15880 consensus_queue.cc:1243] T a50a506e17f747e0a940bc3c982336c6 P 66b8c715b9e6479e9218b08bd3156380 [LEADER]: Peer 6921687ed3814cccb2bd9c606875b9f0 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260202 15:09:12.753971 15910 raft_consensus.cc:515] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.756417 15928 raft_consensus.cc:515] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.812510 15926 raft_consensus.cc:3060] T 65f098e59c0240a3a65cbd15dc358b64 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:12.815773 15721 raft_consensus.cc:493] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:12.847043 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69e93e86b75b464fa4c55fdeb141acea" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
W20260202 15:09:12.841955 15849 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260202 15:09:12.847935 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:12.860481 12717 raft_consensus.cc:3060] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:12.815403 15663 raft_consensus.cc:493] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.862985 15663 raft_consensus.cc:515] T ed220e8ce3ce45ff8b531267a15de19e P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.850256 12720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "435e192b63f348a58c00c3ae9e598ea9" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.851573 12719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0"
I20260202 15:09:12.832641 15673 raft_consensus.cc:3060] T 30df58c1134a49e18c29202abd31da41 P 66b8c715b9e6479e9218b08bd3156380 [term 5 FOLLOWER]: Advancing to term 6
I20260202 15:09:12.877656 12794 raft_consensus.cc:3060] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 3 FOLLOWER]: Advancing to term 4
W20260202 15:09:12.881898 12614 leader_election.cc:336] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191): Timed out: RequestConsensusVote RPC to 127.11.46.196:36191 timed out after 1.648s (SENT)
I20260202 15:09:12.882390 12614 leader_election.cc:304] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 1eca64246cad41379cde4e2f9c3e25d2, 66b8c715b9e6479e9218b08bd3156380
W20260202 15:09:12.847304 15837 consensus_meta.cc:287] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0: Time spent flushing consensus metadata: real 0.695s user 0.006s sys 0.004s
I20260202 15:09:12.839234 15508 raft_consensus.cc:493] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6921687ed3814cccb2bd9c606875b9f0)
I20260202 15:09:12.886938 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01f20099bc6844498a65c943dde6b1ff" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
I20260202 15:09:12.889030 12791 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da327288552c40f59790d9001c7f7e5d" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:12.887569 12793 raft_consensus.cc:2468] T 01f20099bc6844498a65c943dde6b1ff P 1eca64246cad41379cde4e2f9c3e25d2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 8.
I20260202 15:09:12.893405 12717 raft_consensus.cc:2468] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 6.
I20260202 15:09:12.894929 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.896410 12717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b57a31e08c1420ab954d13d66af9297" candidate_uuid: "1eca64246cad41379cde4e2f9c3e25d2" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6921687ed3814cccb2bd9c606875b9f0" is_pre_election: true
I20260202 15:09:12.896986 12794 raft_consensus.cc:2468] T 69e93e86b75b464fa4c55fdeb141acea P 1eca64246cad41379cde4e2f9c3e25d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf140958c3b4f339c512a9dc4be2dab in term 4.
I20260202 15:09:12.897816 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d12e74920dc474599bb20f8e977a6b4" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2"
I20260202 15:09:12.898301 12717 raft_consensus.cc:1275] T 62adc5e06e9d428f94bdbd03c58c5e5b P 6921687ed3814cccb2bd9c606875b9f0 [term 5 FOLLOWER]: Refusing update from remote peer 1eca64246cad41379cde4e2f9c3e25d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260202 15:09:12.898715 12794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe02793fdaa448caa739180083258aae" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:12.899019 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 223114us
W20260202 15:09:12.901684 12615 leader_election.cc:336] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367): Timed out: RequestConsensusVote RPC to 127.11.46.194:44367 timed out after 1.936s (SENT)
I20260202 15:09:12.902047 12615 leader_election.cc:304] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf140958c3b4f339c512a9dc4be2dab; no voters: 66b8c715b9e6479e9218b08bd3156380, 6921687ed3814cccb2bd9c606875b9f0
W20260202 15:09:12.904686 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 145262us
I20260202 15:09:12.907173 15608 raft_consensus.cc:2749] T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:12.858099 15721 raft_consensus.cc:3060] T 4687c40305f64dd2960fcbb336f93395 P 1eca64246cad41379cde4e2f9c3e25d2 [term 6 FOLLOWER]: Advancing to term 7
I20260202 15:09:12.913710 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4687c40305f64dd2960fcbb336f93395" candidate_uuid: "bbf140958c3b4f339c512a9dc4be2dab" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1eca64246cad41379cde4e2f9c3e25d2" is_pre_election: true
W20260202 15:09:12.877988 15920 log.cc:927] Time spent T 3b57a31e08c1420ab954d13d66af9297 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20260202 15:09:12.832530 15800 raft_consensus.cc:515] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
I20260202 15:09:12.910763 12725 raft_consensus.cc:2393] T 83fc6cc808d9450788aeecc86caa94bd P 6921687ed3814cccb2bd9c606875b9f0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eca64246cad41379cde4e2f9c3e25d2 in current term 9: Already voted for candidate 6921687ed3814cccb2bd9c606875b9f0 in this term.
W20260202 15:09:12.927241 13326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 106221us
I20260202 15:09:12.928678 15941 raft_consensus.cc:493] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.929143 15955 consensus_queue.cc:1059] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.151s
W20260202 15:09:12.888307 15919 log.cc:927] Time spent T d91ac50607b34cd1b7d61cd42f7c26d8 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260202 15:09:12.889437 15540 raft_consensus.cc:2749] T d88e30bfa6e641749640dde238a2a8f8 P bbf140958c3b4f339c512a9dc4be2dab [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260202 15:09:12.819413 15935 raft_consensus.cc:515] T 6ca9793f4e4e46418e44fbdc3d3a4794 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196" port: 36191 } attrs { promote: false } }
W20260202 15:09:12.957329 12614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.196:36191, user_credentials={real_user=slave}} blocked reactor thread for 75469.4us
W20260202 15:09:12.959017 13327 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.959996 15943 raft_consensus.cc:2749] T cb4cc1899a25452c9c57398fe4022645 P bbf140958c3b4f339c512a9dc4be2dab [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260202 15:09:12.961313 15959 log.cc:927] Time spent T e4d8302d577042feaa44f63d596dc87d P bbf140958c3b4f339c512a9dc4be2dab: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260202 15:09:12.934939 15804 raft_consensus.cc:493] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:12.926806 15957 raft_consensus.cc:2804] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260202 15:09:12.903162 15837 raft_consensus.cc:515] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
W20260202 15:09:12.982234 12616 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:12.995579 13329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260202 15:09:12.997475 12764 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260202 15:09:12.977039 15540 raft_consensus.cc:2749] T 9e78bd8179024ae0aca19878b91342a9 P bbf140958c3b4f339c512a9dc4be2dab [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260202 15:09:12.935220 15508 raft_consensus.cc:515] T 4d68e0d1324241f1878c450b36fac925 P 1eca64246cad41379cde4e2f9c3e25d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:12.981745 15957 raft_consensus.cc:493] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:09:13.017692 12724 raft_consensus.cc:1240] T a739ed9c340d4dd9935861456bc3f3d3 P 6921687ed3814cccb2bd9c606875b9f0 [term 8 FOLLOWER]: Rejecting Update request from peer 1eca64246cad41379cde4e2f9c3e25d2 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260202 15:09:13.020030 12690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.195:45829, user_credentials={real_user=slave}} blocked reactor thread for 114639us
I20260202 15:09:13.022569 12690 leader_election.cc:304] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921687ed3814cccb2bd9c606875b9f0; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
I20260202 15:09:12.980115 15791 leader_election.cc:290] T 4d68e0d1324241f1878c450b36fac925 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 8 election: Requested vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829)
I20260202 15:09:12.925765 15706 consensus_queue.cc:260] T dbc759e31af04c60b10baa9dce36e825 P 1eca64246cad41379cde4e2f9c3e25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:13.036533 15955 raft_consensus.cc:3055] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Stepping down as leader of term 7
I20260202 15:09:12.968271 15941 raft_consensus.cc:515] T 6fc895893bbe4569bb328cfc3513de38 P bbf140958c3b4f339c512a9dc4be2dab [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:13.057605 13326 leader_election.cc:304] T d2476071e1d24aa6a1342c8d81699cef P 66b8c715b9e6479e9218b08bd3156380 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b8c715b9e6479e9218b08bd3156380; no voters: 1eca64246cad41379cde4e2f9c3e25d2, bbf140958c3b4f339c512a9dc4be2dab
W20260202 15:09:13.055423 12688 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.46.193:40057, user_credentials={real_user=slave}} blocked reactor thread for 288535us
W20260202 15:09:13.057493 12763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.000s sys 0.010s
W20260202 15:09:13.060546 12763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.000s sys 0.010s
I20260202 15:09:12.998093 15804 raft_consensus.cc:515] T 62adc5e06e9d428f94bdbd03c58c5e5b P bbf140958c3b4f339c512a9dc4be2dab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "1eca64246cad41379cde4e2f9c3e25d2" member_type: VOTER last_known_addr { host: "127.11.46.195" port: 45829 } }
I20260202 15:09:13.063235 12644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83fc6cc808d9450788aeecc86caa94bd" candidate_uuid: "6921687ed3814cccb2bd9c606875b9f0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bbf140958c3b4f339c512a9dc4be2dab"
I20260202 15:09:13.023516 15957 raft_consensus.cc:3060] T 3b57a31e08c1420ab954d13d66af9297 P 1eca64246cad41379cde4e2f9c3e25d2 [term 9 FOLLOWER]: Advancing to term 10
I20260202 15:09:13.055012 15955 raft_consensus.cc:740] T 9592591cd602455aa52fdc1d84e48b63 P 6921687ed3814cccb2bd9c606875b9f0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6921687ed3814cccb2bd9c606875b9f0, State: Running, Role: LEADER
W20260202 15:09:13.079731 15743 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.246s user 0.032s sys 0.040s
I20260202 15:09:13.079306 15831 raft_consensus.cc:2749] T 0d12e74920dc474599bb20f8e977a6b4 P 6921687ed3814cccb2bd9c606875b9f0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260202 15:09:13.077418 15910 leader_election.cc:290] T 00b1a15d2d1246e1abe52586ddf512ef P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6921687ed3814cccb2bd9c606875b9f0 (127.11.46.194:44367), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
W20260202 15:09:13.096508 15947 log.cc:927] Time spent T 6fc895893bbe4569bb328cfc3513de38 P 6921687ed3814cccb2bd9c606875b9f0: Append to log took a long time: real 0.322s user 0.001s sys 0.000s
I20260202 15:09:13.096776 15800 leader_election.cc:290] T 2ac5476ea30a46548db9ede417d946e6 P bbf140958c3b4f339c512a9dc4be2dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1eca64246cad41379cde4e2f9c3e25d2 (127.11.46.195:45829), 66b8c715b9e6479e9218b08bd3156380 (127.11.46.196:36191)
I20260202 15:09:13.097419 15877 raft_consensus.cc:3055] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 9 LEADER]: Stepping down as leader of term 9
I20260202 15:09:13.097744 15877 raft_consensus.cc:740] T 323d926990494ad1aafb6aa0282d9f68 P 66b8c715b9e6479e9218b08bd3156380 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 66b8c715b9e6479e9218b08bd3156380, State: Running, Role: LEADER
W20260202 15:09:13.101384 12615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.46.194:44367, user_credentials={real_user=slave}} blocked reactor thread for 199704us
I20260202 15:09:13.102833 15938 raft_consensus.cc:493] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:09:13.103299 15938 raft_consensus.cc:515] T 11a10c0c64914444a6659e661207c082 P 66b8c715b9e6479e9218b08bd3156380 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf140958c3b4f339c512a9dc4be2dab" member_type: VOTER last_known_addr { host: "127.11.46.193" port: 40057 } } peers { permanent_uuid: "6921687ed3814cccb2bd9c606875b9f0" member_type: VOTER last_known_addr { host: "127.11.46.194" port: 44367 } } peers { permanent_uuid: "66b8c715b9e6479e9218b08bd3156380" member_type: VOTER last_known_addr { host: "127.11.46.196"